Method and apparatus for subject identification

ABSTRACT

Comprehensive 2D learning images are collected for learning subjects. Standardized 2D gallery images of many gallery subjects are collected, one per gallery subject. A 2D query image of a query subject is collected, of arbitrary viewing aspect, illumination, etc. 3D learning models, 3D gallery models, and a 3D query model are determined from the learning, gallery, and query images. A transform is determined for the selected learning model and each gallery model that yields or approximates the query image. The transform is at least partly 3D, such as 3D illumination transfer or 3D orientation alignment. The transform is applied to each gallery model so that the transformed gallery models more closely resemble the query model. 2D transformed gallery images are produced from the transformed gallery models, and are compared against the 2D query image to identify whether the query subject is also any of the gallery subjects.

CLAIM OF PRIORITY

This application claims the benefit of U.S. Provisional PatentApplication No. 61/920,349, entitled “METHOD AND APPARATUS FOR SUBJECTIDENTIFICATION,” filed Dec. 23, 2013, which is expressly incorporated byreference herein.

FIELD OF THE INVENTION

The present invention relates to subject identification. Moreparticularly, the present invention relates to the identification ofsubjects such as human faces in uncontrolled query images, by extractinga three-dimensional learning model from two-dimensional images andmanipulating that three-dimensional models to account for imagingconditions of the query image.

DESCRIPTION OF RELATED ART

One possible approach for machine implemented subject recognition wouldutilize a gallery of images of various known or already-identifiedsubjects, comparing a query image of an unknown subject against thegallery images.

However, image features may vary from one image to another. For example,considering a human face as the subject, the pose and/or orientation,level(s) and direction(s) of illumination, the direction from which theimage is taken, the portion of the face that is within the frame, etc.may not necessarily be uniform among multiple images. Available galleryimages may not represent a particular query image sufficiently so as tofacilitate reliable identification of a face in the query image.

Modifying or “warping” a gallery image in order to approximate the imageproperties of a query image may not always be possible and/or useful.For example, a gallery image with a frontal view of a subject's face maynot include sufficient information to identify a subject whose faceappears in profile in a query image. In this example, a frontal viewmight not include the side of the head, a clear view of the ears, theback of the neck, etc. Absent sufficient information, a warp that issufficient to turn a frontal image into a profile may yield substantialinaccuracies and/or alterations, which also may impact the reliabilityof identification.

BRIEF SUMMARY OF THE INVENTION

The present invention contemplates a variety of systems, apparatus,methods, and paradigms for subject identification.

In one embodiment of the present invention, a method is provided, themethod including establishing at least one substantially threedimensional learning model of at least one learning subject,establishing at least one substantially three dimensional gallery modelfor at least one gallery subject, and establishing at least onesubstantially three dimensional query model of a query subject. Themethod includes determining a transform of at least one parent gallerymodel from among the gallery models in combination with at least oneactive learning model from among the learning models so as to yield atleast one transformed gallery model, wherein the transformed gallerymodel approaches correspondence with at least one of the query models inat least one model property as compared with the parent gallery model.The method further includes applying the transform, and comparing atleast one substantially two dimensional transformed gallery image atleast substantially corresponding with the transformed gallery modelagainst at least one substantially two dimensional query image at leastsubstantially corresponding with the query model, so as to determinewhether the query subject is gallery subject.

The learning images may include unique state of image properties ascompared with the remainder of the learning images. Those imageproperties may include viewing aspect, illumination, texture, and/orconfiguration.

Each of the gallery images may include at least one substantiallysimilar image property as compared with the remainder of the galleryimages. That image property may include viewing aspect, illumination,texture, and/or configuration.

Each of the query images may include a unique state of image propertiesas compared with the gallery images. Those image properties may includeviewing aspect, illumination, texture, and/or configuration.

The method may include determining the transformed gallery image fromthe transformed gallery model. The method may include determining thequery image from the query model.

Establishing the learning model may include laser scanning, threedimensional tomography, time-of-flight measurement, depth imaging,ultrasonic mapping, holographic imaging, and/or plenoptic photography.Establishing the gallery model may include laser scanning, threedimensional tomography, time-of-flight measurement, depth imaging,ultrasonic mapping, holographic imaging, and/or plenoptic photography.Establishing the query model may include laser scanning, threedimensional tomography, time-of-flight measurement, depth imaging,ultrasonic mapping, holographic imaging, and/or plenoptic photography.

The method may include establishing at least one substantially twodimensional learning image of the at least one learning subject, anddetermining the learning model therefrom. Establishing the learningimage may include digital photography, analog photography, twodimensional scanning, visible light imaging, near infrared imaging,thermal infrared imaging, ultraviolet imaging, monochrome imaging, colorimaging, multispectral imaging, hyperspectral imaging, millimeter waveimaging, transmissive x-ray imaging, and/or backscatter x-ray imaging.

The method may include establishing at least one substantially twodimensional gallery image of the at least one gallery subject, anddetermining the gallery model therefrom. Establishing the gallery imagemay include digital photography, analog photography, two dimensionalscanning, visible light imaging, near infrared imaging, thermal infraredimaging, ultraviolet imaging, monocrhome imaging, color imaging,multispectral imaging, hyperspectral imaging, millimeter wave imaging,transmissive x-ray imaging, and/or backscatter x-ray imaging.

The method may include establishing at least one substantially twodimensional query image of the query subject, and determining the querymodel therefrom. Establishing the query image may include at least oneof digital photography, analog photography, two dimensional scanning,visible light imaging, near infrared imaging, thermal infrared imaging,ultraviolet imaging, monocrhome imaging, color imaging, multispectralimaging, hyperspectral imaging, millimeter wave imaging, transmissivex-ray imaging, and/or backscatter x-ray imaging.

The learning subject may include a human face. The gallery subject mayinclude a human face. The query subject may include a human face.

The learning subject may include a human, an animal, a plant, alandscape feature, a vehicle, a weapon, a food item, and/or a tool. Thegallery subject may include a human, an animal, a plant, a landscapefeature, a vehicle, a weapon, a food item, and/or a tool. The querysubject may include a human, an animal, a plant, a landscape feature, avehicle, a weapon, a food item, and/or a tool.

The method may include determining a pre-transform of at least oneparent query model from among the query models in combination with atleast one active learning model from among the learning models so as toyield at least one transformed query model, wherein the transformedquery model approaches correspondence with at least one of the gallerymodels in at least one model property as compared with the parent querymodel. The method also may include determining the transform as being atleast substantially an inverse of the pre-transform.

The transform may be at least partially a three dimensional transform.The transform may include a three dimensional illumination transfer. Thetransform may include a three dimensional aspect alignment. Thetransform may include a three dimensional reconfiguration. The at leastone model property may include texture, shape, illumination, and/orconfiguration.

In another embodiment of the present invention, a method is provided,the method including establishing at least one substantially threedimensional learning model of at least one learning subject,establishing at least one substantially three dimensional gallery modelfor at least one gallery subject, and establishing at least onesubstantially three dimensional query model of a query subject. Themethod includes determining a transform of at least one parent querymodel from among query models in combination with at least one activelearning model from among the learning models so as to yield at leastone transformed query model, wherein the transformed query modelapproaches correspondence with at least one of the gallery models in atleast one model property as compared with the parent query model. Themethod also includes applying the transform, and comparing at least onesubstantially two dimensional transformed query image at leastsubstantially corresponding with the at least one transformed querymodel against at least one substantially two dimensional gallery imageat least substantially corresponding with the at least one gallerymodel, so as to determine whether the query subject is the gallerysubject.

In another embodiment of the present invention, a method is provided,the method including establishing at least one substantially threedimensional learning model of at least one learning subject,establishing at least one substantially three dimensional gallery modelfor at least one gallery subject, and establishing at least onesubstantially three dimensional query model of a query subject. Themethod includes determining a transform of at least one parent gallerymodel from among the gallery models in combination with at least oneactive learning model from among the learning models so as to yield atleast one transformed gallery model, wherein the transformed gallerymodel approaches correspondence with at least one of the query models inat least one model property as compared with the parent gallery model.The method also includes applying the transform, and comparing thetransformed gallery model against the query model, so as to determinewhether the query subject is the gallery subject.

In another embodiment of the present invention, a method is provided,the method including establishing at least one substantially threedimensional learning model of at least one learning subject,establishing at least one substantially three dimensional gallery modelfor at least one gallery subject, and establishing at least onesubstantially three dimensional query model of a query subject. Themethod includes determining a transform of at least one parent querymodel from among the query models in combination with at least oneactive learning model from among the learning models so as to yield atleast one transformed query model, wherein the transformed query modelapproaches correspondence with at least one of the gallery models in atleast one model property as compared with the parent query model. Themethod also includes applying the transform, and comparing thetransformed query model against the gallery model, so as to determinewhether the query subject is the gallery subject.

In another embodiment of the present invention, a method is provided,the method including capturing a plurality of two dimensional digitallearning images of a learning face, each of the learning imagesincluding a unique state of viewing aspect, illumination, texture, andconfiguration as compared with a remainder of the learning images, anddetermining computationally a three dimensional learning model from thelearning images. The method includes capturing a plurality of twodimensional digital gallery images, one gallery image from each of aplurality of gallery faces, each of the gallery images including a stateof at least substantially similar viewing aspect, illumination, andconfiguration as compared with a remainder of the gallery images, anddetermining computationally a plurality of three dimensional gallerymodels from the gallery images, one for each of the plurality of galleryfaces. The method includes capturing a two-dimensional query image of aquery face, the query image including a state of viewing aspect,illumination, and configuration at least substantially different fromany of the gallery images, and determining computationally a threedimensional query model from the query image. The method also includesdetermining for each of the gallery models a pre-transform of the querymodel in combination with the learning model so as to yield atransformed query model, wherein each transformed query model approachescorrespondence with regard to at least one of texture, shape,illumination, and configuration with a respective one of the gallerymodels, as compared with the query model, and determining for each ofthe gallery models a transform as being at least substantially aninverse of the respective pre-transform therefor. The method furtherincludes applying the transforms to the respective gallery models so asto yield transformed gallery models, determining computationally a twodimensional transformed gallery image from each of the transformedgallery models, and comparing each of the transformed gallery imagesagainst the query image so as to determine whether the at least onequery subject is any of the gallery subjects.

In another embodiment of the present invention, an apparatus isprovided, the apparatus including a processor, and at least one of asensor, a data store, and a communicator, in communication with theprocessor. The apparatus includes a learning image establisher includingexecutable instructions instantiated on the processor, the learningimage establisher being adapted to establish at least one twodimensional learning image of at least one learning subject via thesensor, the data store, and/or the communicator. The apparatus includesa learning model determiner including executable instructionsinstantiated on the processor, the learning model determiner beingadapted to determine at least one three dimensional learning model fromthe learning images. The apparatus includes a gallery image establisherincluding executable instructions instantiated on the processor, thegallery image establisher being adapted to establish at least one twodimensional gallery image of at least one gallery subject via thesensor, the data store, and/or the communicator. The apparatus includesa gallery model determiner including executable instructionsinstantiated on the processor, the gallery model determiner beingadapted to determine at least one three dimensional gallery model fromthe gallery images. The apparatus includes a query image establisherincluding executable instructions instantiated on the processor, thequery image establisher being adapted to establish at least one twodimensional query image of at least one query subject via the sensor,the data store, and/or the communicator. The apparatus includes a querymodel determiner including executable instructions instantiated on theprocessor, the query model determiner being adapted to determine atleast one three dimensional query model from the query images. Theapparatus includes a learning model selector including executableinstructions instantiated on the processor, the learning model selectorbeing adapted to select at least one active learning model from thelearning models. The apparatus includes a pre-transform determinerincluding executable instructions instantiated on the processor, thepre-transform determiner being adapted to determine a pre-transform ofat least one parent query model from among the query models incombination with at least one active learning model from among thelearning models so as to yield at least one transformed query model,wherein the transformed query model approaches correspondence with atleast one of the gallery models in at least one model property ascompared with the parent query model. The apparatus includes a transformdeterminer including executable instructions instantiated on theprocessor, the transform determiner being adapted to determine thetransform as being at least substantially an inverse of thepre-transform. The apparatus includes a model transformer includingexecutable instructions instantiated on the processor, the modeltransformer being adapted to transform the at least one gallery model toyield at least one transformed gallery model. The apparatus includes atransformed gallery image determiner including executable instructionsinstantiated on the processor, the transformed gallery image determinerbeing adapted to determine at least one two dimensional transformedgallery image from the transformed gallery models. The apparatusincludes an image comparer including executable instructionsinstantiated on the processor, the image comparer being adapted tocompare the transformed gallery images against the query images so as todetermine whether the query subject is the gallery subject.

In another embodiment of the present invention, an apparatus isprovided, the apparatus including a processor, a sensor in communicationwith the processor, the sensor being adapted to sense two dimensionalimages, and at least one of a data store and a communicator, also incommunication with the processor. The apparatus includes a learningmodel establisher including executable instructions instantiated on theprocessor, the learning model establisher being adapted to establish atleast one three dimensional learning model of at least one learningsubject via the data store and/or communicator. The apparatus includes agallery model establisher including executable instructions instantiatedon the processor, the gallery model establisher being adapted toestablish at least one three dimensional gallery image of at least onegallery subject via the data store and/or the communicator. Theapparatus includes a query image establisher including executableinstructions instantiated on the processor, the query image establisherbeing adapted to establish at least one two dimensional query image ofat least one query subject via the sensor. The apparatus includes aquery model determiner including executable instructions instantiated onthe processor, the query model determiner being adapted to determine atleast one three dimensional query model from the query images. Theapparatus includes a learning model selector including executableinstructions instantiated on the processor, the learning model selectorbeing adapted to select at least one active learning model from amongthe learning models. The apparatus includes a pre-transform determinerincluding executable instructions instantiated on the processor, thepre-transform determiner being adapted to determine a pre-transform ofat least one parent query model from among the query models incombination with at least one active learning model from among thelearning models so as to yield at least one transformed query model,wherein the transformed query model approaches correspondence with atleast one of the gallery models in at least one model property ascompared with the parent query model. The apparatus includes a transformdeterminer including executable instructions instantiated on theprocessor, the transform determiner being adapted to determine thetransform as being at least substantially an inverse of thepre-transform. The apparatus includes a model transformer includingexecutable instructions instantiated on the processor, the modeltransformer being adapted to transform the gallery models to yield atleast one transformed gallery model. The apparatus includes atransformed gallery image determiner including executable instructionsinstantiated on the processor, the transformed gallery image determinerbeing adapted to determine at least one two dimensional transformedgallery image from the transformed gallery models. The apparatusincludes an image comparer including executable instructionsinstantiated on the processor, the image comparer being adapted tocompare the transformed gallery images against the query images so as todetermine whether the query subject is the gallery subject.

In another embodiment of the present invention, a head mounted displayis provided, the head mounted display including a processor, a sensor incommunication with the processor, the sensor being adapted to sense twodimensional images, and a data store and/or a communicator incommunication with the processor. The head mounted display includes alearning model establisher including executable instructionsinstantiated on the processor, the learning model establisher beingadapted to establish a three dimensional learning model of a learningsubject via the data store and/or the communicator. The head mounteddisplay includes a gallery model establisher including executableinstructions instantiated on the processor, the gallery modelestablisher being adapted to establish a three dimensional gallery imageof each of a plurality of gallery subjects via the data store and/or thecommunicator. The head mounted display includes a query imageestablisher including executable instructions instantiated on theprocessor, the query image establisher being adapted to establish a twodimensional query image of a query subject via the sensor. The headmounted display includes a query model determiner including executableinstructions instantiated on the processor, the query model determinerbeing adapted to determine a three dimensional query model from thequery image. The head mounted display includes a pre-transformdeterminer including executable instructions instantiated on theprocessor, the pre-transform determiner being adapted to determine apre-transform of the query model in combination with the learning modelso as to yield at least one transformed query model approachingcorrespondence with the gallery models in at least one of texture,shape, illumination, and configuration as compared with the parent querymodel. The head mounted display includes a transform determinerincluding executable instructions instantiated on the processor, thetransform determiner being adapted to determine at least one transformas at least substantially an inverse of the pre-transforms. The headmounted display includes a model transformer including executableinstructions instantiated on the processor, the model transformer beingadapted to transform the gallery models to yield a plurality oftransformed gallery models. The head mounted display includes atransformed gallery image determiner including executable instructionsinstantiated on the processor, the transformed gallery image determinerbeing adapted to determine two dimensional transformed gallery imagesfrom the transformed gallery models. The head mounted display includesan image comparer including executable instructions instantiated on theprocessor, the image comparer being adapted to compare the transformedgallery images against the query image so as to determine whether querysubject is any of the gallery subjects. The head mounted displayincludes an outputter in communication with the processor, the outputterbeing adapted to output visual content regarding a comparison result asto whether the query subject is any of the gallery subjects. Theprocessor, the sensor, the at least one of the data store and thecommunicator, and the outputter are disposed on a frame, the frame beingconfigured so as to be wearable on the head of a wearer, wherein whenthe frame is worn the outputter is disposed proximate, facing, andsubstantially aligned with at least one eye of the wearer, and thesensor is disposed so as to at least substantially match a line of sightof at least one eye of the wearer.

In another embodiment of the present invention, an apparatus isprovided, the apparatus including means for establishing at least onesubstantially three dimensional learning model of at least one learningsubject, means for establishing at least one substantially threedimensional gallery model for at least one gallery subject, and meansfor establishing at least one substantially three dimensional querymodel of a query subject. The apparatus includes means for determining atransform of at least one parent gallery model from among the gallerymodels in combination with at least one active learning model from amongthe learning models so as to yield at least one transformed gallerymodel, wherein the transformed gallery model approaches correspondencewith at least one of the query models in at least one model property ascompared with the parent gallery model. The apparatus also includesmeans for applying the transform, and means for comparing at least onesubstantially two dimensional transformed gallery image at leastsubstantially corresponding with the transformed gallery model againstat least one substantially two dimensional query image at leastsubstantially corresponding with the query model, so as to determinewhether the query subject is the gallery subject.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Like reference numbers generally indicate corresponding elements in thefigures.

FIG. 1 illustrates an example embodiment of a method for identifyingfaces in images according to the present invention, in flow chart form.

FIG. 2A and FIG. 2B show example arrangements of imagers and viewingaspects relative to a learning face.

FIG. 3A through FIG. 3F show example views of a learning face fromdifferent viewing aspects according to the present invention.

FIG. 4A and FIG. 4B show example arrangements of imagers and viewingaspects relative to a learning face.

FIG. 5A, FIG. 5B, FIG. 5C, and FIG. 5D show example views of a learningface, a gallery face, a query face, and a transformed query face,respectively, according to the present invention.

FIG. 6A, FIG. 6B, FIG. 6C, and FIG. 6D show example photographsrepresenting a query image, a gallery image, a transform, and atransformed gallery image, respectively, according to the presentinvention.

FIG. 7 shows an example embodiment of a method for identifying faces inimages according to the present invention, incorporating camera captureof images, processor computation of models, and incorporation of apre-transform, in flow chart form.

FIG. 8 shows an example embodiment of a method for identifyingnon-specified subjects according to the present invention, consideringthree dimensional models directly without necessarily determining thosemodels from two dimensional images.

FIG. 9 shows an example embodiment of an apparatus for identifyingsubjects according to the present invention, in schematic form.

FIG. 10A through FIG. 10C show an example embodiment of an apparatus foridentifying subjects according to the present invention as divided intosub-units, in schematic form.

FIG. 11A through FIG. 11C show an example embodiment of an apparatus foridentifying subjects according to the present invention as divided intosub-units, in perspective view.

DETAILED DESCRIPTION OF THE INVENTION

With reference to FIG. 1, an example method according to the presentinvention is described herein. More particularly, FIG. 1 shows in flowchart form an example embodiment of a method for identifying human facesin images according to the present invention.

For clarity, the example method in FIG. 1 addresses a single, concreteclass of subjects, namely human faces. However, the present invention isnot limited only to consideration of human faces. For example, forcertain embodiments consideration of other subjects such as automobiles,firearms, dogs, etc. may be equally suitable. It is also emphasized thatthe present invention is not limited only to one type of subject, e.g. asingle embodiment might address both human faces and automobiles.

Furthermore, for simplicity the example method in FIG. 1 is described toat least some degree in qualitative terms. More detailed and/or concretedescription is presented subsequently herein with regard to otherexamples and/or figures. However, it is emphasized that the presentinvention is not particularly limited with regard to how the inventionmay be implemented in terms of mathematics. Although for at leastcertain embodiments, incorporation of concepts and/or mathematicsrelating to sparse representation and/or three dimensional morphablemodels (“3DMM”) may be useful, this is an example only, and otherarrangements may be equally suitable.

Typically, though not necessarily, a method such as that described withregard to FIG. 1 may be carried out using a processor, for example adigital electronic processor with executable instructions instantiatedthereon. However, this is an example only, and other arrangements may beequally suitable.

In the example arrangement of FIG. 1, multiple two-dimensional learningimages are established 102 depicting a single three-dimensional learningface. Learning images depict, describe, define, etc. the subjectclass—human faces—in general. Learning images may be considered to serveas a baseline for a class or type of subjects. For the example in FIG.1, color digital JPG or PNG format photographs may be taken of a realand living person who is serving as learning subject during aphotography session. As a more particular example, color digital imagesmight be taken of the learning face from many aspects (e.g. front,profile, top, etc.), while the face exhibits different postures (e.g.smiling, frowning, speaking, etc.), and/or under different lightingconditions (e.g. top, front, bottom, tightly focused, diffuse,incandescent, fluorescent, etc.), so as to provide extensive datacharacterizing the learning subject. However, the present invention isnot particularly limited with regard to either the form or properties ofthe learning images and/or how images are established, and otherarrangements may be equally suitable.

With regard to establishing images in the present invention, whetherlearning images or otherwise, the term “establish” should be understoodconsidered broadly. It is noted that to “establish” something may,depending on particulars, refer to either or both the creation ofsomething new (e.g. establishing a business, wherein a new business iscreated) and the determination of a condition that already exists (e.g.establishing the whereabouts of a person, wherein the location of aperson who is already present at that location is discovered, receivedfrom another source, etc.). Similarly, establishing an image mayencompass several potential approaches, including but not limited to thefollowing.

Establishing an image may include capturing images from a physicalentity (e.g. photographing a human face), rendering images from a datamodel, or otherwise producing the image from some parent entity.

Establishing an image also may include creating the image “from scratch”without regard to a parent entity, e.g. a processor may executeinstructions so as to create an image in some fashion, whether fromexisting data, user inputs, internal algorithms, etc.

Establishing an image additionally may include acquiring apreviously-existing image, for example by reading an image file from adata store, downloading an image through a communication link, orotherwise obtaining an image that already exists substantially in a formas to be used by some embodiment of the present invention.

The present invention is not particularly limited insofar as how imagesmay be established. It is required only that an image that is functionalin terms of the present invention is in some fashion made manifest.Other arrangements than those described may be equally suitable. Also,where used with regard to other steps such as establishing a model,etc., the term “establish” should be similarly be interpreted in a broadfashion.

It is emphasized that in the example of FIG. 1, multiple images areestablished 102 of a single face. Typically though not necessarily,learning images are established 102 so as to represent the learning facewith multiple aspects (i.e. as seen from different directions), undermultiple lighting conditions, in multiple poses and/or facialexpressions, etc. For the example of FIG. 1, the learning images serveto characterize a single learning face to a sufficient degree as toprovide a baseline for future consideration of other faces. In morecolloquial terms, the learning images facilitate learning “what a facelooks like”, in at least broad terms. For example, features such as ageneral shape of a human head, the existence, shape, position, etc. ofeyes, nose, mouth, and so forth, ranges of motion of jaw, lips, eyelids,etc. may be documented and made available through suitable learningimages.

As previously noted, the steps in FIG. 1 may be carried out in aprocessor. A processor may not “know” that certain data represent (forexample) head shape, eye position, etc. The description of “learning” asapplied to learning images, learning faces, etc. is descriptive, and thepresent invention is not limited to colloquial understandings thereof.

Continuing in FIG. 1, a three dimensional learning model is determined104 from the two dimensional learning images established in step 102.Typically though not necessarily, the three dimensional learning modelmay be determined 104 computationally, through spatial analysis offeatures within the two dimensional learning images, though otherarrangements may be equally suitable.

The present invention is not particularly limited with regard to whatinformation may be incorporated into the learning model. Typicallythough not necessarily, a learning model may include informationregarding one or more of shape, texture, and illumination of a face.

Shape may be considered as representing the volume, surface, directionof normality, etc. of the face. In some sense the shape, and thuslikewise in some sense the learning model as a whole, might beenvisioned as a three dimensional “sculpture” of the original learningface, as based on the multiple two dimensional learning imagesestablished for that learning face. For at least certain embodiments,this may be at least approximately correct.

However, human faces are not necessarily static with regard to shape.For a particular human face hair may move, the jaw may open and close,muscles may contract and relax, the lips may change shape, eyes may openand shut, etc. Similarly, on longer timescales the shape of a face maychange due to changing hairstyle, injury, aging, etc. Furthermore,elements that may not necessarily be considered part of the face in avery strict sense nevertheless may be significant when considering theface as a shape, such as glasses, hats, jewelry, etc., other “worn” ortemporary features, etc.

Thus, although it may be convenient to refer to “the” shape of a face asbeing a fixed form, in practice a face may not have a single shape.Likewise, although it may be convenient to refer to shape as a singletime-fixed quantity (and for simplicity such reference is made at placesherein), a learning model may not necessarily have a fixed shape.Rather, the learning model may be considered to be variable,“morphable”, etc. Thus a given learning model may not necessarilyrepresent a face only in a single configuration, and/or at a singlemoment in time, but may include a range of possible expressions,hairstyles, etc.

With regard to texture in a learning model, the term “texture” may referto actual texture, e.g. roughness or smoothness of a surface. However,“texture” may refer in addition and/or instead to other surfaceproperties, such as coloring, translucency, reflectivity, etc. It isnoted that terms such as “texture”, “texture map”, etc. are terms of artwith regard to surfaces of three dimensional computer graphical models,and/or other models.

As noted with regard to shape, texture in a human face is notnecessarily static. Color, reflectivity, etc. may change with exertion,emotion, tanning, aging, etc. Also, color notably may be deliberatelyaltered e.g. through application of makeup, use of hair dye, and soforth; other texture properties likewise may be deliberately altered.Texture information in a learning model also may not necessarily berestricted only to a single state or moment in time.

With regard to illumination in a learning model, illumination may referto features such as direction, intensity, spread, color, etc. of a lightsource or light sources illuminating a face (and similarly shadowing .For example, a face lit from directly underneath the chin may appearvisually quite different from a face lit from above, even if no otherfactors change (e.g. the shape, viewing aspect, texture, etc.). Whilelighting, shadowing, etc. may not necessarily be considered part of aface in a strict sense, lighting, and/or response to lightingnevertheless may be incorporated into a learning model for a faceaccording to the present invention.

As noted with regard to shape and texture, lighting of a human face isnot necessarily static. Motion of a face, motion of light sources,changes in the brightness, color, spread, etc. of lighting, changes inconditions affecting light (e.g. fog, dust, smoke, etc. may impactlighting as incident on the face, regardless of whether those conditionsaffect obstruct or otherwise directly affect viewing of the face), etc.may manifest. Lighting in a learning model also may not necessarily berestricted only to a single state or moment in time.

Comments made with regard to content and variability of the learningmodel, and to shape, texture, lighting, and/or other informationtherein, should be understood as applying similarly to other models,such as gallery and query models described subsequently herein. However,this should not be taken to imply that all learning, gallery, and/orquery models must include shape, texture, and/or lighting information,that all learning, gallery, and/or query models must be variable, and/orthat learning models, gallery models, and/or query models must or willinclude identical or similar data to one another. These are examplesonly, and other arrangements may be equally suitable.

Moving on in FIG. 1, a two dimensional gallery image is established 106for each of multiple three dimensional gallery faces. Gallery faces arefaces of various individuals, typically though not necessarily theindividuals against which unknown faces are to be compared. Galleryimages thus may be considered to serve as example images, against whichan unknown query image (described subsequently herein) may be matched,e.g. so as to determine whether a face in a gallery image is the same asan unknown query face in a query image.

For the example in FIG. 1, gallery images may be color digital JPG orPNG format photographs taken of real persons who are serving as gallerysubjects. However, the present invention is not particularly limitedwith regard to either the form or properties of the gallery imagesand/or how the gallery images are established, and other arrangementsmay be equally suitable.

Typically though not necessarily, gallery images may be to at least somedegree standardized. Gallery images may be established so as to havesimilar size, resolution, face posture, lighting properties, focus,aspect (e.g. full-front, left profile, etc.) Such an arrangement may beunderstood as being perhaps similar to standardized database photographssuch as driver's license or passport photographs, wherein faces ofindividuals are imaged under controlled lighting, from a full-frontaspect, with a neutral facial expression, etc. However, the presentinvention is not particularly limited with regard to the specifics ofthe gallery images, and non-uniform gallery images and/or otherarrangements may be equally suitable.

Also typically though not necessarily, gallery faces may be associatedwith additional information. For example, the names, ages, addresses,etc. of the relevant persons may be known. In such an example, the namesand/or other information relating to the gallery faces also may beassociated with the gallery images, either when the gallery images areestablished 106 or subsequently. To continue the example, the name ofeach individual might be attached to a digital file that makes up orincludes the digital image (likewise the individual's age, address,etc.). Other information, including but not limited to informationregarding when, where, under what conditions, etc. the gallery imagesare established 106 similarly may be associated with the gallery facesand/or the gallery images. For example, for gallery images obtained asJPG or PNG format digital image files, the camera make and model, date,time, GPS location, focus, etc. may be so associated, as may beinformation regarding lighting conditions, the operator of the camera(if any), and so forth. For arrangements wherein such information isassociated with some or all of the gallery images, those gallery imagesand/or the gallery faces depicted therein may in some sense beconsidered to be “known” images and/or faces.

However, the present invention is not particularly limited with regardto what information, if any, may be associated with the gallery facesand/or gallery images, and other information or no information at allmay be equally suitable. For example, it is permissible for a galleryimage and/or the associated gallery face to be unknown; thus, a galleryimage of an unknown gallery face might be compared according to thepresent invention with a query image of a query face that also isunknown. While such comparison may not result in identifying the queryface, other functions may be accomplished, for example determiningwhether a person of unknown identity depicted in two different images(one treated as a gallery image and one as a query image) is the sameperson.

It is also noted that the learning face may or may not also be one ofthe gallery faces. While including the learning face as a gallery face(and thus obtaining both learning images and a gallery image of the sameface) is not required, neither is such prohibited.

In addition, it is noted that different learning images, gallery images,etc. may represent a single subject (e.g. a single human face) atdifferent times, and/or otherwise in different states. For example,first and second gallery images and gallery models might depict the sameface from the same aspect, with the same illumination, etc. but whereinthe first gallery image depicts the face at age 20 and the secondgallery image depicts the face at age 50. Such an arrangement may forexample facilitate determination of whether a particular query imagerepresents that face at or near a given time. Different gallery imagesof the same face also might represent the subject while wet or dry, withdifferent hairstyles, etc. Although certain of such states may berepresented within the learning images and/or the learning model, thisdoes not exclude representation of such states within the gallery imagesand/or gallery models in addition or instead.

Still with reference to FIG. 1, a three dimensional gallery model isdetermined 108 for each gallery face from the corresponding twodimensional gallery image for that gallery face, the gallery imageshaving been established in step 106. Typically though not necessarily,the three dimensional gallery models may be determined 108computationally, through spatial analysis of features within the twodimensional gallery images, though other arrangements may be equallysuitable.

As noted with regard to the learning model, it may be convenient toimagine the gallery models as three dimensional “sculptures” of theoriginal gallery faces, as based on the single two dimensional galleryimages obtained for each gallery face. However, as again noted withregard to the learning model, such a convenient description is notnecessarily correct for all embodiments of the present invention, andgallery models may not necessarily be static or limited to a singleconfiguration and/or moment in time. Other arrangements may be equallysuitable.

Typically, though not necessarily, gallery models may include dataregarding shape, texture, and/or illumination.

Typically, though not necessarily, the gallery models may be determined108 so as to incorporate less accuracy, fidelity, total data, etc. thanthe learning model. For example, a three dimensional gallery modeldetermined 108 using data from only one two-dimensional gallery imagemay include less information than a three dimensional learning modeldetermined using data from many two-dimensional learning images. Thus,for at least certain embodiments the gallery models may exhibit lowerresolution, less overall fidelity, less precision, etc. However, this isan example only, and other arrangements may be equally suitable.

Optionally, some or all gallery models may be determined 108 withreference to and/or including information from the learning model. Forexample, consider a learning model that may be a relativelycomprehensive representation of a learning face, by virtue of beingbased on many two dimensional learning images of the learning face (asopposed to gallery models that may for example utilize data from onlyone gallery image). Such a comprehensive learning model may be utilizedto inform the determination of gallery models from single correspondinggallery images. That is, if the learning model may be considered torepresent in at least a general sense “what a human face looks like” interms of shape, motion, etc., then that general sense of the shape,motion, etc. of a human face may be utilized when determining thegallery models. As a concrete example, even if a single gallery image isfull-frontal, without information regarding the shape of features notvisible therein such as the back of the head, a gallery model based onsuch a gallery image may refer to the learning model in determining thatthe back of the head is (or at least to a high probability may be)generally convex in shape, etc. Thus, even though gallery models may bedetermined 108 from only a single gallery image each, the determination108 of gallery models may nevertheless take advantage of informationfrom the learning model to provide more accuracy, precision, fidelity,etc. than may be expected from a model generated only from a single twodimensional image. In more colloquial terms, when modeling a human facefrom a single photo, it may be informative to know generally what ahuman face looks like from other photographs.

However, referring to and/or including information from the learningmodel in determining 108 the gallery models is not required, and otherarrangements may be equally suitable. In addition, for at least certainembodiments the gallery models may not be, and/or may not be required tobe, complete models. For example, a gallery model based on a singleleft-profile image of a gallery face may not include informationregarding the right side of the face, and such information may not berequired.

Still with reference to FIG. 1, to briefly summarize results from steps102 through 108, many two dimensional learning images have been acquiredwith comprehensive information for a single learning face, and adetailed three dimensional learning model has been generated from thatcomprehensive information, such that the learning model may be suitableas a baseline standard for a human face. Further, individual twodimensional gallery images for each of many gallery faces have beenacquired with at least basic information regarding those many galleryfaces, and three dimensional gallery models have generated from thatbasic information, such that the gallery models may serve as possiblecomparisons for unknown faces.

Continuing in FIG. 1, a two dimensional query image is established 110for a three dimensional query face. A query face is the face of aperson, typically though not necessarily a person of unknown identity; aquery image thus represents data that is to be compared with galleryimages, e.g. so as to determine whether the query face is the same asone of the gallery faces.

For the example in FIG. 1, a query image may be a color digital JPG orPNG format photographs taken of some real person. However, the presentinvention is not particularly limited with regard to either the form orproperties of the query image and/or how the query image is established,and other arrangements may be equally suitable.

Typically though not necessarily, learning images and gallery images maybe clear, complete (i.e. showing the full face) ,well-lit, of highresolution, etc. By contrast, query images may frequently be images thatare “whatever is available”: the query face may be partially obscured oronly partially in the field of view, the focus may be poor, theresolution may be low, there may be motion blur, poor lighting, etc.While the present invention does not exclude query images that are ofgood quality, in practice query images frequently may not be of goodquality.

Typically, though not necessarily, the query face and one of the galleryfaces may be the same face, i.e. the same person. However, this is notrequired; if a comparison of a query face against a group of galleryfaces yields no match, this in itself may be useful information, e.g. anindication that whoever the query subject may be, that query subject isnot one of a number of known gallery subjects.

Moving on in FIG. 1, a three dimensional query image is determined 112for the query face from the query image. Typically though notnecessarily, the three dimensional query model may be determined 112computationally, through spatial analysis of features within the twodimensional query image, though other arrangements may be equallysuitable.

As noted with regard to the learning and gallery models, it may beconvenient to imagine the query model as a three dimensional “sculpture”of the original query face, as based on the single two dimensional queryimage obtained for the query face. However, as again noted with regardto the learning and gallery models, such a convenient description is notnecessarily correct for all embodiments of the present invention, andquery models may not necessarily be static or limited to a singleconfiguration and/or moment in time. Other arrangements may be equallysuitable.

Typically, though not necessarily, a query model may include dataregarding shape, texture, and/or illumination.

Also typically though not necessarily, and as already noted with regardto the gallery models, the query model may be determined 112 so as toincorporate less accuracy, fidelity, total data, etc. than the learningmodel. However, other arrangements may be equally suitable.

Also as with the gallery models, optionally the query model may bedetermined 112 with reference to and/or including information from thelearning model. Thus even though the query model may be determined 112from only a single query image, the determination 112 of the query modelmay nevertheless take advantage of information from the learning modelto provide more accuracy, precision, fidelity, etc. than may be expectedfrom a model generated only from a single two dimensional image.

However, referring to and/or including information from the learningmodel in determining 112 the query model is not required, and otherarrangements may be equally suitable. In addition, for at least certainembodiments the query models may not be, and/or may not be required tobe, complete models. For example, a query model based on a singleleft-profile image of a gallery face may not include informationregarding the right side of the face, and such information may not berequired.

Still with reference to FIG. 1, a transform is determined 118 forcombinations of each gallery model with the learning model, wherein theresult approaches correspondence with the query model.

For example, consider an arrangement wherein in the query image, thequery face is lit from a light source above and to the left of the queryface, and the query face is also turned to the right and downwardrelative to the camera (or some other image source), and the query faceexhibits a posture as if speaking. As may be understood, suchparticulars may affect the appearance of the query face in the queryimage, and likewise may affect the query model which was determined fromthe query image.

Consider further that for this particular example, in the gallery imagesthe gallery faces are lit from directly in front, are facing directlyinto the camera (or other image source), and are in a neutral,non-speaking posture. Again, such particulars may affect the appearanceof the gallery faces in the gallery images, and likewise the gallerymodels determined from the gallery images.

Given such an arrangement, the query image may correspond poorly withthe gallery images at least in terms of lighting and viewing aspect, andthe query model likewise may correspond poorly with the gallery models.In more colloquial terms, for purposes of comparison than query imagehas “the wrong lighting”, and is viewed from “the wrong angle”. Althoughthis particular example may be considered arbitrary, in general it maycommonly happen that the query and gallery images and models do notnecessarily reflect corresponding arrangements with regard to how thefaces are oriented, lit, postured, etc.

It is noted that this issue—the likelihood that query images may not berelied upon to reliably correspond with available gallery images—maypose a significant problem in machine-implemented subject recognition.The ability to address this issue with a high degree of reliability andsuccess is also an advantage of the present invention (though notnecessarily the only such advantage). The present invention enablesexecuting subject recognition even when the image that is to berecognized bears little apparent resemblance to the known image used forcomparison, i.e., the query model corresponds poorly with the gallerymodels.

Thus as noted, a transform is determined 118 for each gallery model incombination with the learning model such that the result does at leastsubstantially correspond with the query model.

Typically though not necessarily, the transform determined 118 in FIG. 1is a mathematical transform, with the learning, gallery, and querymodels or at least certain properties thereof expressed in mathematicalform. Approaching correspondence thus may be considered with regard tomathematical similarity, for at least certain embodiments of the presentinvention. In qualitative terms, typically (though not necessarily)approaching correspondence may address features such as viewing aspect,illumination, facial configuration (e.g. jaw and mouth posture, changesin hair growth and style, etc.), texture (e.g. coloring, reflectivity,etc.). Colloquially, the term “approaching correspondence” with regardto the present invention is used herein to indicate an increasedsimilarity in such features, i.e. the viewing aspects are made moresimilar, etc.

However, it is emphasized that embodiments of the present invention maynot require perfect or even near-perfect correspondence, nor is aparticular degree of absolute correspondence necessarily required.Rather, correspondence should be approached, i.e. the degree ofsimilarity should improve. For example, if the query model and aparticular gallery model exhibit different viewing aspects, thetransform may result in those viewing aspects being more closelyaligned.

For certain embodiments, a high degree of correspondence or even anear-perfect or perfect mathematical match between the query image andthe result of the transform as applied to a gallery model and thelearning model may be suitable. For other embodiments, less rigoroussimilarity may be acceptable. The degree of correspondence and/or ofimprovement in correspondence may for example depend upon factors suchas the available data, including but not limited to the data availablein the query image. For example, given a very poor quality query image,with limited and/or uncertain data therein, a very close correspondencemay be impossible, impractical, or problematic. Regardless, the presentinvention is not particularly limited with regard to a degree ofcorrespondence or a degree of improvement of correspondence.

In colloquial terms, the transform serves to adjust the gallery modelsto more closely resemble the query model, so that the gallery faces aspresented therein resemble the query face as presented in the queryimage. That is, the gallery faces are made to “look like” the queryface, so as to facilitate comparison thereof (described subsequentlyherein, for example with regard to step 126).

Continuing in FIG. 1, the transform as determined in step 118 is applied120 to each gallery model, so that the gallery models approachcorrespondence with the query model. Typically, though not necessarily,this may result in changes in lighting, shape, etc. of the gallery facesas represented in the gallery models. Thus, following step 120 thetransformed gallery models more closely correspond with the query model.

Two dimensional transformed gallery images are determined 122 from thethree dimensional transformed gallery models. Typically though notnecessarily, the two dimensional transformed gallery images may bedetermined 122 computationally, through spatial analysis of featureswithin the three dimensional transformed gallery models, though otherarrangements may be equally suitable. This may be (but is not requiredto be) in some sense a reversal of method step 108, wherein threedimensional gallery models were determined from two dimensional galleryimages. Where step 108 addresses determination of three dimensionalmodels from two dimensional images, step 122 addresses determination oftwo dimensional images from three dimensional models. In colloquialterms, steps 108 through 122 may be considered to “up convert” 2Dgallery images and a 2D query image into 3D gallery models and a 3Dquery model, to transform the 3D gallery models to correspond with the3D query model, and then to “down convert” the transformed 3D gallerymodels back into 2D transformed gallery images.

It is noted that in the arrangement shown in FIG. 1, no step is shownwherein the three dimensional query model is used to determine a twodimensional image. Because the query model is not transformed, a twodimensional image determined from the query model would at leastapproximate the original query image, so a conversion from 2D to 3D andback to 2D may not be necessary or useful, at least for certainembodiments of the present invention. While determining a new twodimensional query image is not prohibited, neither is such stepnecessarily required.

Continuing in FIG. 1, the two dimensional query image is compared 126against the two dimensional transformed gallery images, e.g. todetermine whether the query face in the query image is the same as anyof the gallery faces in any of the gallery images.

Following step 126, the method shown in FIG. 1 is complete. However, thepresent invention is not limited only to the steps shown in the examplearrangement of FIG. 1. Other steps and/or repetitions of steps alreadyshown may be equally suitable. Likewise, subdividing steps shown may besuitable for at least certain embodiments, and incorporating a methodaccording to the present invention into a larger method, or breaking amethod according to the present invention into smaller sub-sections,also may be equally suitable.

At this point it may be useful to illustrate graphically certainfeatures of the present invention, including some as may already havebeen referred to with regard to FIG. 1.

Now with reference to FIG. 2A, as noted previously a learning modelaccording to the present invention may be determined from multipleviewing aspects, under multiple lighting conditions, with multiplefacial postures, etc. By way of illustration, in FIG. 2A a learning face232A is shown in top-down view, with several imagers 258A-1 through258A-5 shown therein, arranged so as to capture images of the learningface 232A along several viewing aspects 256A-1 through 256A-5. Imager258A-1 (shown in FIG. 2A as a camera, such as a digital point-and-shootcamera, though this is an example only and other arrangements may beequally suitable) is disposed approximately full-front (as viewed topdown, with full-front at the bottom of edge of FIG. 2A); imager 258A-2is disposed approximately 15 degrees right of full-front; imager 258A-3is disposed approximately 30 degrees right of full-front; imager 258A-4is disposed approximately 45 degrees right of full-front; and imager258A-5 is disposed approximately 45 degrees left of full front. Viewingaspects 256A-1 through 256A-5 are likewise arranged facing inward fromapproximately full-front, 15 degrees right, 30 degrees right, 45 degreesright, and 45 degrees left of full-front, respectively. These areexample aspects and imagers only, presented for illustrative purposes,and other arrangements may be equally suitable.

It is noted that the learning face 232A is shown in FIG. 2A as a subjectof an example embodiment of the present invention, not necessarily as anelement thereof, and that other faces and/or subjects shown anddescribed herein may not necessarily be part of the respective exampleembodiments of the present invention shown and described.

With reference to FIG. 2B, a learning face 232B is shown in left profileview, with several imagers 258B-1 through 258B-3 shown therein, arrangedso as to capture images of the learning face 232B along several viewingaspects 256B-1 through 256B-3. Imager 258B-1 is disposed approximately15 degrees above horizontal; imager 258B-2 is disposed approximately athorizontal; and imager 258B-3 is disposed approximately 15 degrees belowhorizontal. Viewing aspects 256B-1 through 256B-3 are likewise arrangedfacing inward from approximately 15 degrees above, 0 degrees, and 15degrees below horizontal, respectively. These also are example aspectsand imagers only, presented for illustrative purposes, and otherarrangements may be equally suitable.

Now with reference to FIG. 3A through FIG. 3E, therein are shown severalexample views of a learning face, e.g. as might correspond with viewingaspects as shown in FIG. 2A and FIG. 2B. As noted, typically a learningface may be imaged from multiple viewing aspects; the present inventionis not particularly limited with regard to specific viewing aspects, andthose shown in FIG. FIG. 3A through FIG. 3E (and likewise FIG. 2A andFIG. 2B) are examples only.

In FIG. 3A, a learning face 332A is visible therein, with eyes 338A,nose 340A, mouth 342A, ears 344A, and hair 346A thereof also visible.(The learning face 332A is presented in a somewhat simplified andabstracted form rather than being fully realistic, for purposes ofclarity.) As may be seen, the learning face 332A is orientedapproximately full-frontal with no inclination from the horizontal, withrespect to the point of view of FIG. 3A; alternately, it may beconsidered that the viewing aspect is approximately in the horizontaland angled at 0 degrees relative to the learning face 332A.

In addition, a framing box 354A is shown in FIG. 3A. The framing box354A is an artifact, which may not be present for a real face. However,as may be understood from comparison with FIG. 3B through FIG. 3E, theframing box 354A in FIG. A may illustrate a feature relevant to twodimensional images depicting three dimensional subjects, namely apparentdistortion related to viewing aspect.

With regard to viewing aspect, it is noted that the view of the learningface 332A in FIG. 3A may correspond at least approximately to viewingaspect 256A-1 in FIG. 2A, and to viewing aspect 256B-2 in FIG. 2B.

With regard to FIG. 3B, a learning face 332B is also visible therein,with eyes 338B, nose 340B, mouth 342B, ears 344B, and hair 346B thereofalso visible, and a framing box 354B. The learning face 332B is orientedapproximately 15 degrees left with no inclination from horizontal, oralternately the viewing aspect is oriented 15 degrees right with noinclination. As may be seen the framing box 354B is distorted comparedto the framing box 354A in FIG. 3A; although the framing box is in atleast approximately the same position and orientation relative to therespective learning faces, the framing box 354B in FIG. 3B appearsoffset to the left and appears smaller on the left edge thereof ascompared with the framing box 354A in FIG. 3A. With regard to viewingaspect, the view of the learning face 332B in FIG. 3B may correspond atleast approximately to viewing aspect 256A-2 in FIG. 2A, and to viewingaspect 256B-2 in FIG. 2B.

With reference to FIG. 3C, a learning face 332C is visible with eyes338C, nose 340C, mouth 342C, and hair 346C and a framing box 354C.However, only one ear 344C is visible. The learning face 332C isoriented approximately 30 degrees left with no inclination fromhorizontal, or alternately the viewing aspect is oriented 30 degreesright with no inclination. The framing box 354C is again distortedcompared to the framing boxes 354A and 354B in FIG. 3A and FIG. 3B. Theview of the learning face 332C in FIG. 3C may correspond at leastapproximately to viewing aspect 256A-3 in FIG. 2A, and to viewing aspect256B-2 in FIG. 2B.

Now referring to FIG. 3D, a learning face 332D is visible with eyes338D, nose 340D, mouth 342D, one ear 344D, hair 346D, and a framing box354D. The learning face 332D is oriented approximately 45 degrees leftwith no inclination from horizontal, or alternately the viewing aspectis oriented 45 degrees right with no inclination. The framing box 354Dis again distorted compared to framing boxes 354A through 354C in FIG.3A through FIG. 3C. The view of the learning face 332D in FIG. 3D maycorrespond at least approximately to viewing aspect 256A-4 in FIG. 2A,and to viewing aspect 256B-2 in FIG. 2B.

In FIG. 3E, a learning face 332E is visible with eyes 338E, nose 340E,mouth 342E, one ear 344E, hair 346E, and a framing box 354E. Thelearning face 332E is oriented approximately 45 degrees left and 15degrees below horizontal, or alternately the viewing aspect is oriented45 degrees right and 15 degrees above horizontal relative to thelearning face 332E. The framing box 354E is again distorted compared toframing boxes 354A through 354D in FIG. 3A through FIG. 3D. In addition,attention is called to the apparent relative positions of the eyes 338E:one eye (the left of the learning face 332E, on the right side of FIG.3E as shown) appears to be below the other eye. By contrast, in FIG. 3Athe eyes 338A therein appear to be at similar heights to one another.

The view of the learning face 332E in FIG. 3E may correspond at leastapproximately to viewing aspect 256A-4 in FIG. 2A, and to viewing aspect256B-1 in FIG. 2B.

Now with regard to FIG. 3F, a learning face 332F is visible with eyes338F, nose 340F, mouth 342F, one ear 344F (though the opposite ear fromthat visible in FIG. 3E), hair 346F, and a framing box 354F. Thelearning face 332F is oriented approximately 45 degrees right and 15degrees above horizontal, or alternately the viewing aspect is oriented45 degrees left and 15 degrees below horizontal relative to the learningface 332F. The framing box 354F is again distorted compared to framingboxes 354A through 354E in FIG. 3A through FIG. 3E. In addition,attention is again called to the apparent relative positions of the eyes338F: one eye again appears to be below the other eye (the right of thelearning face 332F, on the left side of FIG. 3F as shown). By contrast,in FIG. 3A the eyes 338A therein appear to be at similar heights to oneanother, while in FIG. 3E the relative heights of the eyes 338E appearreversed.

The view of the learning face 332E in FIG. 3E may correspond at leastapproximately to viewing aspect 256A-4 in FIG. 2A, and to viewing aspect256B-1 in FIG. 2B.

Although FIG. 3A through FIG. 3F are of learning faces, certain featuresthat may not necessarily be exclusive to learning faces and/or learningimages according to the present invention may be observed throughcomparison of FIG. 3A through FIG. 3E.

As may be seen from FIG. 3A through FIG. 3E, whether a particularfeature of a three dimensional subject (in the case of FIG. 3A throughFIG. 3E, a human face) is visible in a given two dimensional imagethereof may be at least in part a function of the viewing aspect (oralternately, the orientation of the subject). For example, in FIG. 3Atwo ears 344A are visible, while in FIG. 3C only one ear 344C isvisible. While presumably the learning face 332C in FIG. 3C also mayhave a second ear, that second ear is not visible therein.

As also may be seen from FIG. 3A through FIG. 3E, in particular withregard to the framing boxes 354A through 354E therein, even for featuresof a three dimensional subject (in the case of FIG. 3A through FIG. 3E,a human face) that are visible in two images taken from differentaspects, at least some degree of distortion may be present when viewinga two dimensional image thereof. Notably, the framing boxes 354A through354E are of visibly different shape in FIG. 3A through FIG. 3E. Whilethe framing boxes are, as noted, comparison artifacts that may notnecessarily be present for a real learning face, nevertheless thedistortion thereof may be illustrative of distortion that may be presentwhen considering a two dimensional image of a three dimensional subject.

This variability in visible features and in apparent perspective may beunderstood in that a two dimensional image of a three dimensional objectmay in at least some sense be considered a “projection” of that threedimensional object into two dimensions. Thus, the relative orientationof the three dimensional subject with regard to the viewing aspect of atwo dimensional image thereof may affect both what may be visible andthe apparent shape, size, position, etc. of what is visible.

This variability in appearance of two dimensional images of threedimensional subjects may be of significance in matching such subjects.This variability is described further subsequently herein.

As noted, the arrangements in FIG. 2A and FIG. 2B and in FIG. 3A throughFIG. 3F are examples. Although only a few viewing aspects and imagestherefor are shown in FIG. 2A and FIG. 2B and in FIG. 3A through FIG.3F, more comprehensive image arrangements also may be suitable. Withregard now to FIG. 4A and FIG. 4B, other example arrangements shown, asmay be considered more comprehensive in terms of viewing aspect.

In FIG. 4A, a learning face 432A is shown in a top-down view. An array459A of imagers is shown; while for simplicity the imagers in the imagerarray 459A are not individually identified as in FIG. 2A and FIG. 2B, itmay be understood that as shown the imager array 459A includes multipleimagers at least somewhat similar to those shown and described withregard to FIG. 2A and FIG. 2B. In the example of FIG. 4A, twenty-fourimagers are shown in the array 459A, surrounding the learning face 432Aat intervals of approximately 15 degrees. Similarly, an array 457A ofviewing aspects also is shown, corresponding with the points of view ofthe imagers in the imager array 459A.

In FIG. 4B a learning face 432B is shown in a left profile view. Anotherarray 459B of imagers is shown, the array 459B as shown includingnineteen such imagers disposed around the learning face 432B atintervals of approximately 15 degrees. A gap is left in the array 459Bas shown to accommodate the body of a subject providing the learningface 432B (the body not being shown in FIG. 4B), though this is anexample only and other arrangements may be equally suitable. Inaddition, an array 457B of viewing aspects is shown corresponding withthe points of view of the imagers in the imager array 459A.

More, fewer, and different. Point out that substantially anything thatgenerates a 2D “image” is an option.

The arrangements of imagers and/or viewing aspects as shown in FIG. 4Aand FIG. 4B are examples only, and the present invention is notparticularly limited with regard to how learning images may beestablished for a learning face, and/or along what viewing aspects suchlearning images may be established.

Arrangements having more imagers and/or viewing aspects, fewer imagersand/or viewing aspects, and/or different configurations of imagersand/or viewing aspects may be equally suitable.

Furthermore, although FIG. 4A and FIG. 4B show a one-to-onecorrespondence between imagers and viewing aspects, other arrangementsalso may be suitable. For example, a single imager, and/or severalimagers, may be configured so as to establish learning images fromseveral viewing aspects each. As a more concrete example, a singleimager might be disposed movably, capturing images from many differentviewing aspects in succession as the imager is moved with respect to thelearning face. Conversely, a single imager might be disposed in astationary configuration, with the learning face being moved withrespect to the imager. Some combination of motion of imager(s) and/orlearning face also may be suitable.

Although the imagers shown in FIG. 4A and FIG. 4B are depicted forillustrative purposes as hand-held cameras, such as a digitalpoint-and-shoot camera, this is an example only. For embodiments whereinimaging hardware is utilized, the present invention is not particularlylimited with regard to the specifics of such hardware. Substantially anysystem adapted to establish a two dimensional image may be suitable. (Inaddition, directly establishing three dimensional models, withoutnecessarily first establishing two dimensional images, also may beequally suitable, as is described subsequently herein.)

Similarly, the present invention is not particularly limited as to whatform a two dimensional learning image may take. While color digitalimages may be suitable for certain embodiments, film based and/ormonochrome images also may be suitable. Images in wavelengths other thanvisible light, including but not limited to near infrared, thermalinfrared, millimeter wave, back scatter x-ray, and ultraviolet may beequally suitable. Images not based on light, such as an ultrasoundimage, also may be suitable. Substantially any two dimensional image maybe suitable for use with at least certain embodiments of the presentinvention, and the present invention is not particularly limited withregard thereto.

Furthermore, although FIG. 4A and FIG. 4B and certain other figures anddescriptions herein refer to human faces as subjects, these are examplesonly. The present invention is not limited only to human faces aslearning, gallery, and/or query subjects, and other subjects, whetherliving or otherwise, may be equally suitable.

In addition, although the arrangements in FIG. 4A and FIG. 4B do notaddress sources of illumination, facial posture, and/or facial texture,it may be understood that as described, it may be understood from theexamples shown in FIG. 4A and FIG. 4B with regard to viewing aspect thatlearning images may include information regarding such feature (and/orother features), and that such features also may be varied. For example,as imagers may be arranged along various viewing aspects as shown inFIG. 4A and FIG. 4B, light sources likewise might be arranged alongvarious aspects relative to the learning face. Light sources also mayvary in color, brightness, focus, etc.

The comments made with regard to learning images and the establishingthereof may also apply similarly to gallery images and/or query images,except where otherwise noted herein.

With regard to distinguishing learning, gallery, and query images andmodels, and illustrating model transform and transformed gallery images,FIG. 5A through FIG. 5D are now referenced.

In FIG. 5A, an example learning face 532A is shown. Given the limits oftwo dimensional illustration, FIG. 5A also might be considered torepresent element 532A as depicting (in two dimensions) a threedimensional learning model, and/or as depicting a two dimensionallearning image. Regardless, as may be seen, in FIG. 5A eyes 538A, mouth542A, ears 544A, and hair 546A are visible therein.

Typically, though not necessarily, a learning image corresponding to afull frontal view as in FIG. 5A may be available; as noted, manylearning images may be established, and a full frontal view may be oneof such learning images. Alternately, a learning model may besufficiently detailed that regardless of whether a full frontal image isavailable, a full frontal view nevertheless may be obtained byappropriately orienting the learning model.

In FIG. 5B, an example gallery face 534B is shown. Again, FIG. 5B alsomight be considered to represent element 534B as depicting a threedimensional gallery model, and/or as depicting a two dimensional galleryimage. As may be seen, in FIG. 5B mouth 542B, ears 544B, and hair 546Bare visible therein. However, the hair 546B in FIG. 5B is visiblydifferent in form (hairstyle) than the hair 546A in FIG. 5A. Inaddition, eyes are not directly visible in FIG. 5B, being obscured byglasses 552B. As may be understood, for example in considering the hair546B and glasses 552B in FIG. 5B, the gallery face may be significantlydifferent from the learning face in FIG. 5A.

Typically, though not necessarily, a gallery image corresponding to afull frontal view as in FIG. 5B may be available; at least certainexisting groupings and/or databases of images, such as driver's licensephotographs, passport images, mug shots, etc. may include full frontalviews. Regardless of what precise image(s) and/or viewing aspect(s) areavailable in a gallery, typically (though not necessarily) galleryimages may exhibit at least somewhat uniform appearance, e.g. most orall gallery images have a similar viewing aspect, similar lighting, asimilar facial posture, etc. Thus, some sort of “standard view” may beexpected for gallery images for at least certain embodiments of thepresent invention (though not necessarily the example view shown in FIG.5B).

Now with reference to FIG. 5C, an example query face 536C is shown.Again, FIG. 5B also might be considered to represent element 536C asdepicting a three dimensional query allery model, and/or as depicting atwo dimensional query image. As may be seen, in FIG. 5C eyes 538C, mouth542C, an ear 544C, and hair 546C are visible therein. However, in FIG.5C a mustache 548C and a beard 550C are visible as well. As may beunderstood, the query face may be significantly different from thelearning face in FIG. 5A, and/or may be significantly different fromsome, perhaps most, of the gallery faces. (Though at least one galleryface may be the same face as the query face, assuming a match is presentin the gallery.)

As may be seen, the viewing aspect in FIG. 5C is not a full frontalview. Rather, the query face 536C is turned to the right and downwardwith respect to the point of view. Typically, though not necessarily,query images may be “as found”. For example, if attempting to identifyan individual from an image taken by a security camera, it may notalways be the case that the individual in question will be imaged in afull-frontal view, under good lighting, etc. Thus, query images may beanticipated to exhibit substantially arbitrary viewing aspects,lighting, facial postures, etc. Also, persons in query images may eitherdeliberately or incidentally vary in their appearance as compared togallery images of those same persons. For example, hairstyles change,facial hair is grown or shaved, scars may be acquired, makeup may beworn, jewelry, glasses, clothing, etc. may cover part or all of theface, subjects may have aged since a gallery image was obtained, etc.Thus, the matter of facial recognition may problematic on multiplelevels.

However, according to the present invention the effectiveness of facialidentification (and of object identification more generally) may beimproved through modifying the appearance of gallery faces so as to moreclosely approach correspondence with a query face. In adjusting, forexample, the viewing aspect of the gallery faces, the lighting of thegallery faces, etc. so as to be more similar to the viewing aspect andlighting of the query face, comparison and identification may be mademore reliable and/or more robust.

As noted previously with regard to step 118 in FIG. 1, in at leastcertain embodiments of the present invention this may be accomplished bydetermining a transform of a three dimensional gallery model to approachcorrespondence with a three dimensional query model.

At least in principle, it may be possible to carry out such a transformin two dimensions, that is, by transforming two dimensional galleryimages to resemble to a dimensional query image. However, this may beproblematic in itself, since the two dimensional images in question arethemselves representative of three dimensional faces. A two dimensionalprojection of a three dimensional subject exhibits a relationship thatmay be described as “dense”, in that many different potential threedimensional constructs might yield or at least approximate a given twodimensional projection. In effect, a transform must be made from one ofa wide range of possible faces to correspond with one of another widerange of possible faces.

By contrast, a three dimensional model exhibits a relationship with athree dimensional subject that may be described as “sparse”, in thatthere is at least approximately a one-to-one correspondence betweenmodel and subject. Thus, a transform is made from one possible face tocorrespond to another one possible face.

In practice, factors such as imperfect data and computationallimitations may restrict determination of absolute one-to-onetransforms. Nevertheless, transforms in three dimensions according tothe present invention may enable high accuracy and/or reliability ofmatching, and/or other advantageous features.

Now with reference to FIG. 5D, therein is shown a transformed galleryface 535D. Yet again, FIG. 5B also might be considered to representelement 536C as depicting a three dimensional query allery model, and/oras depicting a two dimensional query image. As may be seen, in FIG. 5Dmouth 542D, an ears 544D, hair 546D, and glasses 552D are visibletherein.

The arrangement in FIG. 5D represents a gallery face (e.g. similar togallery face 534B in FIG. 5B), subsequent to a transform according tothe present invention, such that the transformed gallery face 535D moreclosely corresponds with the query face 536C in FIG. 5C in terms ofviewing aspect. (In practice, other features besides viewing aspect maybe considering instead or in addition, including but not limited tolighting, facial posture, etc.) That is, through visual inspection thetransformed gallery face 535D may be viewed as “the same face” as thegallery face 534B in FIG. 5B. However, again through visual inspectionthe viewing aspect of the transformed gallery face 535D now correspondsmore closely with the viewing aspect of the query face 536C in FIG. 5C.

Consequently, a comparison of the query face 536C in FIG. 5C and thetransformed gallery face 535D in FIG. 5D may be a more “apples toapples” comparison than would be a comparison of the query face 536Cagainst the gallery face 534B.

However, in FIG. 5C a mustache 548C and a beard 550C are visible aswell. As may be understood, the query face may be significantlydifferent from the learning face in FIG. 5A, and/or may be significantlydifferent from some, perhaps most, of the gallery faces. (Though atleast one gallery face may be the same face as the query face, assuminga match is present in the gallery.)

Turning now to FIG. 6A through FIG. 6D, therein another exampleillustrating a transform according to the present invention is shown.Where FIG. 5A through FIG. 5D illustrated a transform with regard toviewing aspect alone, FIG. 6A through FIG. 6D illustrate a transformwith regard to illumination, viewing aspect, and facial pose.

In FIG. 6A, a query image is shown, that query image exhibitingessentially arbitrary pose (e.g. viewing aspect, orientation to animager, etc.) and lighting (e.g. number of light sources, direction,brightness, etc.) for a query face. In the example shown, lighting ismainly to the right side of the subject's face, and the face is tiltedslightly to the subject's right.

In FIG. 6B, a gallery image is shown, exhibiting a vertical, fullfrontal view of a gallery face. Such view and illumination may beexhibited in at least certain galleries, as noted previously, and may betaken to represent “good” or at least “standardized” viewing conditionsand/or image properties.

Notably, FIG. 6A and FIG. 6B exhibit different lighting, orientation,and facial posture.

FIG. 6C is a graphical representation of a transform as may be appliedto the gallery image in FIG. 6B such that a transformed gallery imagewould more closely correspond (e.g. in illumination, viewing aspect,posture, etc.) to the query image in FIG. 6A. In practice the transformmay not be determined as a graphical construct (though such also is notexcluded), but the transform is illustrated here as an example. Also, itis noted that the transform will not necessarily closely resemble eitherthe query image, query model, gallery image, and/or gallery model(though close resemblance also is not excluded). Notably, the mouth inthe transform of FIG. 6C is curved downward more deeply than either thequery image of FIG. 6A or the gallery image of FIG. 6B, the range ofvariation in illumination is more extreme in FIG. 6C than in FIG. 6A or6B, etc. The transform is not required to resemble either the gallerymodel or the query model (or for that matter the learning model) to anyparticular degree, but rather may be considered as set of directions foraltering the gallery model to more closely correspond with the querymodel.

Although the images in FIG. 6A through FIG. 6D are two dimensional bynecessity, it should be understood that the transform may be a threedimensional construct (if the transform is indeed a spatial construct atall; for certain embodiments, a non-graphical transform, e.g. atransform represented by mathematical relations, may be equallysuitable). Likewise, as previously described, the transform may bedetermined through evaluation of a three dimensional query model and athree dimensional gallery model, rather than directly from twodimensional query and gallery images.

For at least certain embodiments, the transform may be referred to forconvenience as an “illumination transfer”, due to the transformincluding (in at least some embodiments) a map for transferring theapparent illumination of the query model to the gallery model. However,transforms for the present invention are not limited to illumination,and should not be understood as so limited unless otherwise specifiedherein.

In FIG. 6D a transformed gallery image is shown. The transformed galleryimage represents a two dimensional projection of the gallery model forthe gallery image in FIG. 6B after the transform in FIG. 6C is appliedthereto. As may be noted, the transformed gallery image in FIG. 6D nowmore closely corresponds with the query image in FIG. 6A, than does thegallery image shown in FIG. 6B. By visual inspection, the illuminationof the gallery face in the transformed gallery image shown in FIG. 6D isstronger from the right side of the query subject's face, betterapproximating the illumination of the query face in the query imageshown in FIG. 6A. Likewise, the mouth pose and head tilt of the galleryface in the transformed gallery image shown in FIG. 6D more closelycorrespond with the query face in the query image shown in FIG. 6A.

Comparison of the query image against gallery images (subsequent to atransform according to the present invention being applied to thosegallery images) thus is facilitated.

Now with reference to FIG. 7, a somewhat more concrete example methodaccording to the present invention is described. In FIG. 7, anarrangement of the multiple two dimensional learning images are captured702 with a camera, for at least one three dimensional learning face. Forpurposes of explanation, the camera may be a digital camera, and thelearning images may be digital images, but this is an example only andother arrangements may be equally suitable. For learning images,including but not limited to digital images, in at least some suchembodiments learning images may be treated mathematically as beingvectors, e.g. by stacking the pixels of the learning images. Thelearning image may include information regarding properties such astexture, illumination, shape, etc. This is an example only, and otherarrangements may be equally suitable.

It is noted that learning images may be captured 702 for two or morelearning faces, though arrangements wherein only one learning face is soimaged may be equally suitable.

Also, as already noted certain steps and processes of the presentinvention may be subdivided. For example, in establishing learningimages as in step 702 of the present invention, this might be consideredas a sequence of sub-steps, e.g. establishing a unique combination ofviewing aspect, illumination, subject configuration, texture conditions(e.g. wet, dry, oiled, made up, etc.) for a given learning image,capturing the learning image with a camera or other sensor, thenestablishing a new unique combination and capturing another learningimage, etc. until the learning subject (face or otherwise) has beenimaged sufficiently to support the function of the particular embodimentof the present invention in question. Likewise, even those sub-steps maybe further divided, e.g. in a loop such as: set viewing aspect, setillumination, capture learning image; update (e.g. through someincremental change) illumination, capture learning image; repeat untilall desired illumination settings for the viewing aspect are captured;update viewing aspect, set illumination, capture learning image; etc. Asmay be understood such sequences may be almost arbitrarily complex,including numerous features besides aspect and illumination, with manydifferent settings or values for each. Such an arrangement (e.g. a givencombination of viewing aspect, illumination, etc.) may be considered tobe a “state” for a learning image, with learning images having uniquestates. That is, different learning images for a given learning subjectmay not show that learning subject with all image properties the same(e.g. the same viewing aspect, lighting, configuration, etc.) as for anyother learning image for that same learning learning subject.

However, these are examples only, and other arrangements may besuitable.

Continuing in FIG. 7, at least one three dimensional learning model iscomputed 704 in a processor (e.g. determined computationally byexecuting executable instructions instantiated on a processor to processimage data from the learning images also instantiated on the processor),utilizing the learning images. Typically though not necessarily,learning images captured 702 of each learning face may be treatedcollectively with regard to computing 704 one learning model therefrom;that is, all images of learning face one may be used to form a singlemodel of learning face one, while all images of learning face two may beused to form a single model of learning face two. However, otherarrangements may be equally suitable, including but not limited toexcluding certain images, considering certain images of one learningsubject when computing a learning model representing a differentsubject, etc.

Still with reference to step 704 in FIG. 7, consideration of multiplelearning faces (multiple learning image sets, multiple learning models,etc.) may be advantageous in at least certain embodiments of the presentinvention. For example, different learning models might be determinedfor different genders, ethnicities, age ranges, etc. Individual learningmodels thus might be selected as being particularly well-suited tocertain query faces, for example, if the query face is known to be (orsuspected to be) a female in the age range of 12 to 15 years (e.g. for asearch in response to an “Amber alert”), considering specifically alearning model based on a learning face corresponding to that gender andage range may facilitate greater confidence in identification of thequery face, may reduce computational requirements, etc. However, whileconsideration of a single specific learning model, or a sub-set of alarger group of learning models, is permitted according to the presentinvention, such is not required for all embodiments.

Moving on in FIG. 7, at least one gallery image is captured 706 with acamera, for multiple gallery faces. The camera may be a digital camera,and the gallery images may be digital images, but this is an exampleonly and other arrangements may be equally suitable. Typically thoughnot necessarily the gallery images are of at least substantially uniformviewing aspect, illumination, facial posture, etc. For example, fullfrontal images with strong illumination and neutral facial postures suchas may be established for driver's licenses, passports, etc. may besuitable, though the present invention is not limited only thereto.

Again, and as noted with regard to step 702, other steps and processesof the present invention may be subdivided. For example, in establishinggallery images as in step 706, such step may be broken into severalsubsteps, e.g. establish a standardized viewing aspect, establish astandardized illumination, establish a standardized configuration (e.g.facial posture), etc., capture the gallery image, then repeat for othergallery subjects (e.g. faces). This likewise may apply to other steps.In addition, with regard to image states (e.g. values for differentimage properties such as viewing aspect, illumination, etc.) galleryimages may be at least somewhat similar states, and/or at least someimage properties that are similar among gallery images. However, theseare examples only, and other arrangements may be suitable.

A gallery model is computed 708 in a processor for each gallery face,utilizing the gallery images (e.g. determined computationally byexecuting executable instructions instantiated on a processor to processimage data from the gallery images also instantiated on the processor).For arrangements wherein only one gallery image is obtained for eachgallery face, each individual gallery model likewise may be computedfrom only a single image. Regardless of the number of gallery imagesused to compute a given gallery model, learning images and/or learningmodels may inform the computation of some or all of the gallery models.

Typically, though not necessarily, the number of gallery faces, and thusthe number of gallery models, may be large, e.g. hundreds, thousands,millions, etc. However, the present invention is not particularlylimited with regard to how many gallery faces may be considered, and/orhow many gallery models may be computed. For purposes of the example inFIG. 7 multiple gallery faces and gallery models are considered, but inother embodiments only a single gallery face and/or a single gallerymodel may be considered. Also, as noted previously, each gallery facemay be represented by only one gallery image

Still referring to FIG. 7, at least one query image is captured 710 witha camera, for multiple gallery faces. The camera may be a digitalcamera, and the gallery images may be digital images, but this is anexample only and other arrangements may be equally suitable.

It is noted that the camera(s) capturing the query image(s) may notnecessarily be the same camera(s) capturing the learning and/or galleryimages, nor are the camera(s) capturing the learning and/or galleryimages necessarily the same camera(s). Though a single camera capturinglearning, gallery, and/or query images is not excluded, in practicedifferent cameras may capture some or all of each different type ofimage (learning, gallery, and query). For example, learning images maybe captured with cameras incorporated into an imaging station at aresearch site, gallery images captured with cameras at a Department ofMotor Vehicles, and query images with surveillance cameras, televisioncameras, phone cameras, pocket cameras, wearable cameras such as may beincorporated into a head mounted display, etc.

Typically though not necessarily the query images may be of arbitrary or“as obtained” condition with regard to viewing aspect, illumination,facial posture, etc. That is, considering a “state” of image propertiesfor a query image, the state and/or at least some image properties ofthe query image may not be the same as any state and/or image propertiesof the gallery images and/or learning images. More concretely, a queryimage may not have similar viewing aspect, similar illumination, similarconfiguration (e.g. facial posture), etc. of any of the gallery images,or even any of the (potentially more comprehensive) learning images.Rather, a query image captured with a camera on a head mounted displaymight be captured with ambient lighting, from whatever viewing aspecthappens to exist between camera and query face, with whatever expressionthe query face may have at the time, etc.

Moving on in FIG. 7, a query model is computed 712 in a processor foreach query face, utilizing the query images (e.g. determinedcomputationally by executing executable instructions instantiated on aprocessor to process image data from the query images also instantiatedon the processor). For arrangements wherein only one query image isobtained for the query face, the query model likewise may be computedfrom only a single image. Regardless of the number of query images usedto compute a given gallery model, learning images and/or learning modelsmay inform the computation of some or all of the query models.

It is noted that the processor(s) computing the query model(s) may notnecessarily be the same processor(s) computing the learning and/orgallery models, nor are the processor(s) computing the learning and/orgallery models necessarily the same processors(s). Though a singleprocessor computing learning, gallery, and/or query models is notexcluded, in practice different processors may compute some or all ofeach different type of image (learning, gallery, and query). Forexample, learning models may be computed with processors associated withan imaging station at a research site, gallery models computed withprocessors at a Department of Motor Vehicles, and query models withprocessors proximate and/or incorporated into surveillance systems,phones, pocket cameras, processors incorporated into a head mounteddisplay, etc.

Again as previously noted, each query face may be represented by onlyone query image.

Typically only a single query face, and thus a single set of queryimages and a single query model, may be considered at once. For example,one unknown face at a time may modeled, transformed, etc. so as tofacilitate identification against known faces, rather than matching manyunknown faces together to many known faces. However, the presentinvention is not particularly limited in this regard, and otherarrangements may be equally suitable.

Moving on in FIG. 7, at least one of the three dimensional learningmodels (computed in step 704) is selected 714 as an active learningmodel. As noted, multiple learning models may be computed 704, andcertain advantages may attach to considering certain learning modelsand/or not considering other learning models (e.g. considering alearning model more particular to a given query face may produce morereliable results, excluding a learning model clearly unrelated to agiven query face may reduce processing requirements, etc.).

The present invention is not particularly limited with regard to themanner in which active learning models are selected 714 from amongavailable learning models. Such selection may be carried out by aperson, e.g. viewing a query image and judging that the query subject isfemale and thus manually selecting one or more query models associatedwith female faces and/or facial characteristics. Alternately, suchselection might be automated in some fashion, carried out throughcomputations performed on a processor with executable instructionsinstantiated thereon. Other arrangements also may be equally suitable.

If only one learning model is computed 704, then typically that onelearning model may be considered to be selected 714 by default, or step714 may be skipped. Likewise, if all available learning models areconsidered for a given query model, step 714 may be skipped.

Still with reference to FIG. 7, a pre-transform is calculated 716 of thequery model with the active learning model (for purposes of simplicity,it is assumed that only a single query model and active learning modelare considered in this example, though multiple query models and/ormultiple active learning models may be equally suitable), wherein thepre-transform when applied to the query model causes the query model (astransformed) to approach correspondence with each gallery model.

Although for simplicity, certain examples herein refer to “a transform”,in practice the transform (and/or a pre-transform, as in the examplepresented here) may include multiple steps and/or operations. Forexample, transferring illumination between query and gallery models (sothat the transformed gallery model has illumination similar to the querymodel, or vice versa) may be carried out as a separate operation fromaligning viewing aspect between query and gallery models, and conformingpose between query and gallery models may be yet a separate operation.

For purposes of explanation, examples are provided herein describingsteps for a pre-transform that includes two major steps: an exampleillumination transfer, and a subsequent example viewing aspect alignmentincorporating aspects of the illumination transfer (such that thetransform taken together considers both illumination and viewingaspect). It may be understood that similar transforms may also considerfacial posture adjustment, and/or other features. However, these areexamples only, and the present invention is not limited only thereto.

It is emphasized that according to the present invention, regardless ofthe particulars of how the pre-transform and/or transform is carried out(e.g. what mathematical or other approaches are utilized), at least aportion of the pre-transform and/or transform is carried out in threedimensions, with respect to three dimensional models. Thus, transferringillumination may be carried out with regard to three dimensions,alignment may be carried out with regard to three dimensions, etc.

Indeed, carrying out such pre-transform/transform operations in threedimensions—illumination transfer, alignment, etc.—is a notable advantageof the present invention, along with sparse correspondence enabledthrough the consideration of three dimensional models as already noted.However, although these examples may be advantages of the presentinvention, they are not necessarily the only advantages of the presentinvention.

The various operations described with regard to step 716 may be forexample determined computationally by executing executable instructionsinstantiated on a processor. However other arrangements may be equallysuitable.

Moving on in FIG. 7, a transform is computed 718 (e.g. determinedcomputationally by executing executable instructions instantiated on aprocessor) that is the inverse of each pre-transform. That is, if (as inthis example) one or more pre-transforms are computed first that causethe query model to approach correspondence with the gallery models, thenan inverse of such transforms typically may cause the gallery models toapproach correspondence with the query models.

Pre-transforms for causing a query model to approach correspondence thegallery models (rather than causing gallery models to approachcorrespondence with a query model) may be advantageous in at leastcertain embodiments. For example, it may be more convenient (e.g. interms of mathematics and/or computation) to determine transforms thatshare a common starting state (the query model) with many end states(the gallery models), rather than determining transforms that have manystarting states (the gallery models) but that reach a common end state(the query model).

However, this is an example only, and determining pre-transforms andtransforms that are inverses thereof is not necessarily required for allembodiments. Other arrangements may be equally suitable.

Still with reference to FIG. 7, the transforms are applied 720 to thegallery models, such that the gallery models approach correspondencewith the query model. The transforms may be so applied for examplecomputationally by executing executable instructions instantiated on aprocessor, though the present invention is not particularly limited withregard to how the transforms are applied.

With the transforms applied to the original gallery models, one or moretransformed gallery models may be available.

Continuing in FIG. 7, a two dimensional transformed gallery image iscomputed 722 from each three dimensional transformed gallery model, e.g.by executing executable instructions instantiated on a processor, thoughthe present invention is not particularly limited with regard to how thetransformed gallery images are computed.

The two dimensional query image is then compared 726 against the twodimensional transformed gallery images, e.g. to determine whether thequery face in the query image is the same as one of the gallery faces inthe transformed gallery images. The comparison may be carried outcomputationally, for example by executing executable instructionsinstantiated on a processor, though the present invention is notparticularly limited with regard to how the comparison is performed.

In practice, the query image may not be, and is not required to be, anexact match for any of the transformed gallery images. Rather, the queryimage may approximate one or more of the transformed gallery images. Thepresent invention is not particularly limited with regard to a degree towhich the query image must match any of the transformed gallery imagesin order to be considered to identify the query face and a gallery faceas being “the same face”, nor does the present invention evennecessarily require that any such determination be made. For at leastcertain embodiments of the present invention, confidence levels may bedetermined and/or associated with certain matches or potential matches,e.g. the query image matches one of the transformed gallery images to aconfidence of 99.9% (or 99%, 98%, 97.5%, 95%, 75%, etc.). Furthermore,multiple matches may be suitable for at least certain embodiments of thepresent invention (even when two different transformed gallery imagesrepresent two different gallery faces).

Numerous variations regarding features and parameters of the presentinvention may be suitable. Certain such variations have already beennoted, but additional comments regarding some variations may beilluminating. Not all variations are necessarily presented herein, noris the present invention limited only to the variations specificallydescribed.

With reference to FIG. 8, therein is shown another example methodaccording to the present invention. Where the arrangements in FIG. 1 andFIG. 7 assumed that inputs for learning, gallery, and query faces wouldbe in the form of two dimensional images, the present invention is notlimited only thereto. As shown and described with regard to thearrangement of FIG. 8, at least some inputs for learning, gallery,and/or query subjects may be in the form of three dimensional models,without necessarily determining those three dimensional models from twodimensional images.

Any of the three dimensional learning, gallery, and query models may beestablished as three dimensional models, and/or may be determined fromtwo dimensional learning, gallery, and query images respectively, in anycombination. Thus, in certain embodiments of the present invention athree learning dimensional learning model may be established as a threedimensional model, e.g. by three dimensional laser scanning and/or otherthree dimensional scanning processes of a learning face and/or otherlearning subject, while in the same embodiments the three dimensionalgallery and/or query models are determined from two dimensional galleryand/or learning images respectively, the gallery and/or learning imagesbeing established e.g. by digital photography and/or other twodimensional imaging processes of a gallery face or other gallery subjectand/or a query face and/or other query subject.

In addition, where FIG. 1 and FIG. 7 refer specifically to human facesas learning, gallery, and query subjects, the present invention islimited only to human faces as subjects. Other subjects, including butnot limited to animals, physical objects such as firearms, blades, otherweapons, bullets, cars and other vehicle, non-human animals, plants,landscapes, trees, rock formations, buildings, and/or elements orportions of any such, may be equally suitable.

In FIG. 8, at least one three dimensional learning model is established804 of at least one learning subject The present invention is notparticularly limited with regard to how the learning model may beestablished. Where in examples of FIG. 1 and FIG. 7 the threedimensional learning model(s) are determined (e.g. computationally)based on two dimensional learning images, the present invention is notlimited only to determining three dimensional learning models from twodimensional learning images. For example, a three dimensional learningmodel may be established through laser scanning, three dimensionaltomography, time-of-flight measurements, ultrasonic mapping, holographicimaging, plenoptic photography, etc. Alternately, a three dimensionallearning model may be constructed computationally (e.g. throughexecutable instructions instantiated on a processor), without utilizingdirect imaging or other sensing of a living human.

Learning models have been described previously herein. To reiteratebriefly, a learning model provides a model of what a subject and/or aclass of subjects (e.g. faces, firearms, automobiles, etc.) “should looklike”, and may include information including but not limited to texture,shape, illumination, and/or configuration, and variations thereof (e.g.with regard to texture a face or automobile may be modeled exhibitingboth dry and wet surface conditions, while with regard to configurationan automobile may be modeled exhibiting both opened and closed doors,etc.).

These are examples only, and other arrangements may be equally suitable.

At least one three dimensional gallery model is established 808 of atleast one three dimensional gallery subject. The present invention isnot particularly limited with regard to how the gallery model(s) may beestablished, as with establishment of the learning model in step 804.Gallery models have been described previously herein. To reiteratebriefly, a gallery model represents a “target” for comparison, typicallythough not necessarily representing a particular example of a subject orclass of subjects. For example, considering firearms as a subject, agallery model may represent M1918 Browning Automatic Rifles (or moreparticularly the M1918A2 sub-type, or even a specific individualweapon), enabling determination as to whether some other image and/ormodel also represents an M1918 BAR. These are examples only, and otherarrangements may be equally suitable.

Still with reference to FIG. 8, at last one three dimensional querymodel of a three dimensional query subject is established 812. Thepresent invention is not particularly limited with regard to how thequery model may be established, as with the learning and gallery modelsin steps 804 and 808. Query models have been described previouslyherein. To reiterate briefly, a gallery model represents the subject tobe compared with and/or identified from the gallery and/or learningmodels. Typically though not necessarily the query model may be“unknown”, in the sense that while the query subject represented by thequery model may be recognizable as a human face (or an automobile, anaircraft, etc.) the identity of that query subject may not be known.These are examples only, and other arrangements may be equally suitable.

At least one active learning model is selected 814 from among thelearning models. This step may be at least somewhat similar to step 714in FIG. 7, already described herein.

Again with reference to FIG. 8, pre-transforms are determined 816 forthe query model with the active learning model(s) to approachcorrespondence with the gallery models. Transforms that represent theinverses of the pre-transforms are then determined 818, the transformsenabling the gallery models with the active learning model(s) toapproach correspondence with the query model. The transforms are applied820 to the gallery models to approach correspondence with the querymodel. These steps may be at least somewhat similar to steps 716, 718,and 720 in FIG. 7, already described herein.

Two dimensional transformed gallery images are determined 822 from thethree dimensional transformed gallery models. This step may be at leastsomewhat similar to step 722 in FIG. 7, already described herein. It isnoted that in the example of FIG. 7, the three dimensional gallerymodels themselves were computed from two dimensional gallery images, anddetermining the two dimensional transformed gallery images may beconsidered to represent a reversal of the earlier two dimensional tothree dimensional computation. By contrast, in the example of FIG. 8,step 822 does not necessarily represent a reversal of a two dimensionalto three dimensional determination, since the three dimensional gallerymodels where not themselves necessarily determined from two dimensionalgallery images.

Continuing in FIG. 8, a two dimensional query image is determined 824from the three dimensional query model.

Typically though not necessarily, the two dimensional query image may bedetermined 824 computationally, through spatial analysis of featureswithin the three dimensional query model, though other arrangements maybe equally suitable. However, the present invention is not particularlylimited with regard to how the two dimensional query image may bedetermined 824.

The two dimensional query image is compared 826 against the twodimensional transformed gallery images, e.g. to determine whether thequery subject is the same as any of the gallery subjects.

Although the example of FIG. 8 shows determination of two dimensionaltransformed gallery images and a two dimensional query image in steps822 and 824 respectively, and comparison of the two dimensional queryimage against the two dimensional transformed gallery images in step826, the present invention is not limited only to two dimensionalcomparison (nor are two dimensional images necessarily required to bedetermined, in particular though not only for arrangements wherein twodimensional comparison does not take place). For example, in certainembodiments the three dimensional query model may be directly comparedagainst the three dimensional transformed query models, withoutnecessarily producing respective two dimensional images therefrom. Thismay be useful for example in an arrangement (as in FIG. 8) wherein thethree dimensional query model is established without necessarilyutilizing a two dimensional query image. That is, there may be no“original” two dimensional query image, and so the comparison may bemade using the original three dimensional query model.

In addition, with regard to the example of FIG. 8 overall, it may beseen therefrom that conversion between two dimensional images and threedimensional models may for at least certain embodiments of the presentinvention be optional. Furthermore, although FIG. 7 shows all of the twodimensional learning, gallery, and query models being computed from twodimensional learning, gallery, and query images respectively, and FIG. 8shows an arrangement with none of the three dimensional learning,gallery, and query models being determined from two dimensional images,the present invention is not limited to “either or”, and variationsthereon also may be suitable. For example, an embodiment wherein a threedimensional learning model is established without utilizing twodimensional learning images, but the three dimensional gallery and querymodels are determined from two dimensional gallery and query imagesrespectively, also may be suitable.

Now with reference to FIG. 9, an example apparatus 960 according to thepresent invention is shown therein, in schematic form. Certain functionsand/or features relating to the apparatus 960 have been explainedpreviously herein, e.g. with regard to example methods in FIG. 1, FIG.7, and FIG. 8, and this description is not necessarily repeated withregard to FIG. 9.

In the example shown in FIG. 9, the apparatus 960 includes a processor962 adapted to execute instructions instantiated thereon. A wide rangeof processors 960 may be suitable for use with the present invention,and the present invention is not particularly limited with regardthereto. Suitable processors may include, but are not limited to,digital electronic processors. Processors may be physical and/or virtual(e.g. “cloud” processors), may be dedicated and/or general purpose, andmay be either unitary or multiple (e.g. several physical processorsoperating in parallel). Other arrangements also may be equally suitable.

The apparatus also includes an imager 964 adapted to establish twodimensional images, in communication with the processor 962. Asillustrated, the imager 964 is represented as a camera such as a digitalcamera, but this is an example only and the present invention is notlimited only thereto.

The apparatus 960 as shown in FIG. 9 includes a light 966 adapted toproduce and/or direct illumination, in communication with the processor962. As illustrated, the light 966 is represented as a directional lamp,but this is an example only and the present invention is not limitedonly thereto.

The imager 964 and light 966 may be optional for at least certainembodiments of the present invention. If two dimensional images (and/orthree dimensional models absent two dimensional images) and illuminationtherefor may be established alternately, the imager 964 and/or light 966may be omitted.

Still with reference to FIG. 9, the apparatus 960 as shown thereinincludes a data store 968 and a communicator 970 in communication withthe processor 962. The data store is adapted to store data, includingbut not limited to two dimensional images, three dimensional models, andexecutable instructions adapted to be instantiated on the processor 962.The communicator is adapted to send and/or receive data, e.g. forcommunication to and/or from the processor 962, including but notlimited to two dimensional images, three dimensional models, andexecutable instructions adapted to be instantiated on the processor 962.The present invention is not particularly limited with regard tosuitable data stores 968 and/or communicators 970. Suitable data stores968 may include but are not limited to hard drives, solid state drives,and virtual storage such as cloud memory. Suitable communicators mayinclude but are not limited to wired and/or wireless connectors.Suitable communicators also may include input devices such as keyboardsfor keyed input, microphones for voice input, sensors for gestural,positional, and/or motion-related input, etc. The data store 968 and/orcommunicator 970 may be optional for at least certain embodiments, butmay be useful in other embodiments for storing and/or communicating twodimensional learning, gallery, and/or query images, three dimensionallearning, gallery, and/or query models, executable instructions, etc.

The apparatus 960 as shown in FIG. 9 also includes an outputter 972. Asillustrated in FIG. 9, the outputter 972 is represented as a visualdisplay or “screen”, but this is an example only and the presentinvention is not limited only thereto. The outputter 972 may be adaptedto output information so as to be sensed by a user, viewer, etc. of theapparatus 960, or some other person or entity. Such information mayinclude, but is not limited to, two dimensional learning, gallery, andquery images, three dimensional learning, gallery, and query models,and/or information relating thereto. The outputter 972 may be optionalfor at least certain embodiments of the present invention, butnevertheless may be useful for others.

An apparatus 960 according to the present invention is not limited onlyto those elements shown in FIG. 9, and other elements may be present.

Still with reference to FIG. 9, the processor 962 shown therein includesdata entities 974 through 996 disposed thereon, e.g. as associations ofexecutable instructions and/or data instantiated upon the processor 962.

Although in the example of FIG. 9 elements 974 through 996 are all shownto be disposed on a single processor 962, this is an example only. Otherarrangements wherein analogs to some or all of elements 974 to 996 arepresent on each of several processors also may be equally suitable(including possible duplication of such elements on multipleprocessors); one such example is shown subsequently herein in FIG. 10Athrough FIG. 10C.

Moreover, although elements 974 through 996 are shown divided intospecific and discrete units in FIG. 9, this is an example only, and ispresented at least in part for clarity. So long as the functions of thepresent invention are enabled, elements 974 through 996 may be combined,further subdivided, etc.

Continuing with reference to FIG. 9, elements 974 through 996 as showninclude a learning image establisher 974, a learning model determiner976, a gallery image establisher 978, a gallery model determiner 980, aquery image establisher 982, a query model determiner 984, a learningmodel selector 986, a pre-transform determiner 988, a transformdeterminer 990, a model transformer 992, a transformed gallery imagedeterminer 994, and an image comparer 996. As noted previously, theseelements 974 through 996 are presented in FIG. 9 as data entitiesinstantiated on the processor 962, though other arrangements may beequally suitable.

The learning image establisher 974 is adapted to establish at least onetwo dimensional learning image, e.g. by obtaining learning images fromthe imager 964, receiving learning images from the communicator 970,reading learning images from the data store 968, etc. Learning imageshave been previously described herein.

The learning model determiner 976 is adapted to determine at least onethree dimensional learning model from two dimensional learning images,e.g. computationally within the processor 962. Learning models have beenpreviously described herein.

The gallery image establisher 978 is adapted to establish at least onetwo dimensional gallery image, e.g. by obtaining gallery images from theimager 964, receiving gallery images from the communicator 970, readinggallery images from the data store 968, etc. Gallery images have beenpreviously described herein.

The gallery model determiner 980 is adapted to determine at least onethree dimensional gallery model from two dimensional gallery images,e.g. computationally within the processor 962. Gallery models have beenpreviously described herein.

The query image establisher 982 is adapted to establish at least one twodimensional query image, e.g. by obtaining query images from the imager964, receiving query images from the communicator 970, reading queryimages from the data store 968, etc. Query images have been previouslydescribed herein.

The query model determiner 984 is adapted to determine at least onethree dimensional query model from two dimensional query images, e.g.computationally within the processor 962. Query models have beenpreviously described herein.

The learning model selector 986 is adapted to select at least one activelearning model from among three dimensional learning models, e.g.computationally within the processor 962, through input delivered viathe communicator 970, etc. Active learning models have been previouslydescribed herein.

The pre-transform determiner 988 is adapted to determine a pre-transformfor a query model with an active learning model to approachcorrespondence with the gallery models, e.g. computationally within theprocessor 962. That is, the query model if subject to the pre-transformwould more closely correspond with the gallery models in terms of atleast one parameter (e.g. texture, shape, illumination, posture, etc.)than if not subject to the pre-transform. Typically the pre-transform isand/or includes portions that are three-dimensional, e.g. a threedimensional illumination transfer, a three dimensional spatialalignment, etc. Pre-transforms have been previously described herein.

The transform determiner 990 is adapted to determine transforms forgallery models with active learning models to approach correspondencewith a query model, e.g. computationally within the processor 962. Thatis, the gallery models if subject to the transform would more closelycorrespond with the query model in terms of at least one parameter (e.g.texture, shape, illumination, posture, etc.) than if not subject to thetransform. Typically the transform is and/or includes portions that arethree-dimensional, e.g. a three dimensional illumination transfer, athree dimensional spatial alignment, etc. Transforms may be an inverseof pre-transforms, and have been previously described herein.

The model transformer 992 is adapted to apply transforms to gallerymodels such that the gallery models (as transformed) approachcorrespondence with a query model. Transforms and application oftransforms to three dimensional models have been previously describedherein.

The transformed gallery image determiner 994 is adapted to determine twodimensional transformed gallery images from the three dimensionaltransformed gallery models (i.e. the gallery models after the transformis applied thereto). Transformed gallery images have been previouslydescribed herein.

The image comparer 996 is adapted to compare a two dimensional queryimage against two dimensional transformed gallery images. Suchcomparison has been previously described herein.

As has been noted, although FIG. 9 shows an integral apparatus 960according to the present invention, it also may be equally suitable forother embodiments of the present invention for elements and/or groups ofelements to be distinct and/or separated in time, space, etc. from oneanother. For example, learning images may be established and learningmodels determined at one place and time by one set of elements, galleryimages established and gallery models determined at another place andtime by another set of elements, etc.

An example arrangement of such a non-integral apparatus according to thepresent invention is shown in FIG. 10A through FIG. 10C, in schematicform. Therein, apparatuses 1060A, 1060B, and 1060C are shown, and areadapted to interact cooperatively. Elements 1060A, 1060B, and 1060C arenumbered similarly, but uniquely; although it may be considered thatelements 1060A, 1060B, and 1060C are all part of a single apparatus, itmay be equally suitable to consider elements 1060A, 1060B, and 1060C asbeing distinct apparatuses, which may but are not required to cooperate.

With reference to FIG. 10A, an apparatus 1060A is shown thereinincluding a processor 1062A, with an imager 1064A, a light 1066A, a datastore 1068A, and a communicator 1070A in communication with theprocessor 1062A. Two data entities 1074 and 1076 are shown disposed onthe processor 1062A, specifically a learning image establisher 1074 anda learning model determiner 1076.

As may be understood, the apparatus 1060A in FIG. 10A thus may beadapted collectively to acquire two dimensional learning images of somelearning subject (e.g. a human face), and process those images togenerate three dimensional learning models. As a more concrete example,the apparatus 1060A may be an imaging station at a research facility,though this is an example only and other arrangements may be equallysuitable.

With reference now to FIG. 10B, an apparatus 1060B is shown thereinincluding a processor 1062B, with an imager 1064B, a light 1066B, a datastore 1068B, and a communicator 1070B in communication with theprocessor 1062B. Two data entities 1078 and 1080 are shown disposed onthe processor 1062B, specifically a gallery image establisher 1076 and agallery model determiner 1080.

As may be understood, the apparatus 1060B in FIG. 10B thus may beadapted collectively to acquire two dimensional gallery images ofgallery subjects (e.g. a human faces), and process those images togenerate three dimensional learning models. As a more concrete example,the apparatus 1060B may be a photography desk at a Department of MotorVehicles, though this is an example only and other arrangements may beequally suitable.

With reference to FIG. 10C, an apparatus 1060C is shown thereinincluding a processor 1062C, with an imager 1064C, a light 1066C, a datastore 1068C, a communicator 1070C, and an outputter 1072C incommunication with the processor 1062C. (As noted previously, at leastcertain elements of apparatus according to the present invention may beoptional. For example, where the apparatus 1060C in FIG. 10C includes anoutputter 1072C, apparatuses 1060A and 1060B in FIG. 10A and FIG. 10Brespectively do not.) Data entities 1082 through 1096 are shown disposedon the processor 1062C, specifically a query image establisher 1082, aquery model determiner 1084, a learning model selector 1086, apre-transform determiner 1088, a transform determiner 1090, a modeltransformer 1092, a transformed gallery image determiner 1094, and animage comparer 1096.

As may be understood, the apparatus 1060C in FIG. 10C thus may beadapted collectively to acquire two dimensional query images of somequery subject (e.g. a human face), and process those images to generatethree dimensional query models. As a more concrete example, theapparatus 1060C may be a wearable device such as a head mounted display,although this is an example only and other arrangements may be equallysuitable.

Turning now to FIG. 11A through FIG. 11C, an example arrangement of anon-integral apparatus according to the present invention is shown inperspective views. The arrangements shown in perspective view in FIG.11A through FIG. 11C may to at least some degree parallel thearrangements shown in schematic form in FIG. 10A through FIG. 10C.

With reference to FIG. 11A, an apparatus 1160A is shown therein. Alaptop computer 1161A is shown; although not directly visible inperspective view, typically a laptop computer 1161A such as shown mayinclude features such as a processor, a data store, a communicator(keyboard, touch pad, wifi card, etc.), an outputter (display screen),etc. Though also not visible, a processor in the laptop computer 1161Amay support instantiated thereon data entities such as a learning imageestablisher and a learning model determiner.

Also in FIG. 11A, an imaging array 1165A with imagers thereon (theimagers being shown but not individually numbered in FIG. 11A) arrangedin an arc to facilitate capture of two dimensional learning images in avertical arc, the array 1165A also being rotatable to facilitate captureof two dimensional learning images in a horizontal arc; such an arraymay enable capture of two dimensional learning images from a widevariety of viewing aspects in an efficient manner (and may for examplebe controlled by a processor in the laptop computer 1161A).

FIG. 11A also shows a lighting array 1167A of lights (again shown butnot individually numbered), disposed so as to produce illumination froma variety of directions. A learning subject 1132A depicted as a humanface is shown in FIG. 11A for explanatory purposes (e.g. to show asubject as may be imaged with the relative to the imaging array 1165Aand the lighting array 1167A); however, the learning subject 1132Ashould not be interpreted as being an integral part of an apparatusaccording to the present invention.

As may be understood, the apparatus 1160A in FIG. 11A thus may beadapted collectively to acquire two dimensional learning images of somelearning subject (e.g. a human face), and process those images togenerate three dimensional learning models. Such an apparatus 1160A mayfor example be disposed at an imaging site at a research facility,though this is an example only and other arrangements may be equallysuitable.

Now with reference to FIG. 11B, an apparatus 1160B is shown therein. Alaptop computer 1161B is shown; although not directly visible inperspective view, typically a laptop computer 1161B such as shown mayinclude features such as a processor, a data store, a communicator(keyboard, touch pad, wifi card, etc.), an outputter (display screen),etc. Though also not visible, a processor in the laptop computer 1161Bmay support instantiated thereon data entities such as a gallery imageestablisher and a gallery model determiner.

Also in FIG. 11B, an imager 1164B is shown arranged so as to capture afull frontal two dimensional gallery image of a gallery subject 1134B.Two lights 1166B are shown arranged so as to provide frontal/overheadlighting from left and right of the gallery subject 1134B. It is notedthat the gallery subject subject 1134B is shown in FIG. 11B forexplanatory purposes (e.g. to show a subject as may be imaged with therelative to the imager 1164B and the lights 1166B); however, the gallerysubject 1134B should not be interpreted as being an integral part of anapparatus according to the present invention.

As may be understood, the apparatus 1160B in FIG. 11BA thus may beadapted collectively to acquire two dimensional gallery images ofgallery subjects (e.g. human faces), and process those images togenerate three dimensional gallery models. Such an apparatus 1160B mayfor example be disposed at a Department of Motor Vehicles office, thoughthis is an example only and other arrangements may be equally suitable.

Now with reference to FIG. 11C, an apparatus 1160C is shown therein. Theexample apparatus 1160C is shown in the form of a head mounted display,though this is an example only and other arrangements may be equallysuitable.

The apparatus 1160C includes a frame 1198C resembling a pair of glasses.A processor 1162C is disposed on the frame 1198C. Although not visiblein perspective view, the processor 1198C may support instantiatedthereon data entities such as a query image establisher, a query modeldeterminer, a learning model selector, a pre-transform determiner, atransform determiner, a model transformer, a transformed gallery imagedeterminer, and an image comparer.

The apparatus 1160C shown in FIG. 11C also includes two imagers 1164C,as may be adapted to capture two dimensional query images, and twooutputters 1172C. The imagers 1164C and outputters 1172C are engagedwith the frame 1198C such that if the apparatus 1160C is worn, theimagers 1164C may be aimed so as to at least substantially be alignedwith the wearer's line of sight and/or field of view; and such that theoutputters 1172C may be disposed facing, proximate, and substantiallyaligned with the viewer's eyes. However such arrangements are examplesonly, and other arrangements may be equally suitable.

No query subject is shown in FIG. 11C, though as may be understood withthe apparatus 1160C worn, the wearer may arrange the apparatus 1160Csuch that the imagers 1164C may capture query images of a query subject.

As may be understood, the apparatus 1160C in FIG. 11C thus may beadapted collectively to acquire two dimensional query images of somequery subject (e.g. a human face), and process those images to generatethree dimensional query models. As a head mounted display, such anapparatus 1160C may enable recognition of faces in the wearer's field ofview, for example so as to support augmented reality by displayingnames, reminders, and/or other information relating to persons, objects,and so forth. This is an example only, and the present invention is notlimited only thereto.

With regard to FIG. 9, FIG. 10A through FIG. 10C, and FIG. 11A throughFIG. 11C, it is noted that the example apparatuses shown therein areadapted to establish two dimensional images and determine threedimensional models therefrom. However, as previously noted, othervariations, including but not limited to establishing three dimensionalmodels without recourse to two dimensional images, also may be equallysuitable. Accordingly, embodiments lacking such elements as the learningimage establisher, gallery image establisher, and/or query imageestablisher may be equally suitable, along with other variations inkeeping with the present invention.

In addition, although FIG. 10A through FIG. 10C and FIG. 11A throughFIG. 11C show specific divisions and arrangements of elements, thepresent invention is not limited only to such divisions and/orarrangements. Other arrangements may be equally suitable. For example,in certain embodiments of the present invention the processor(s) may bedistinct from the imager(s), with imager(s) simply capturing images(e.g. learning images, gallery images, query images), and otherfunctions such as determining three dimensional models, determiningpre-transforms and/or transforms, etc. being performed in a processordistinct and potentially separated from the imager(s) by substantialamounts of distance and/or time.

Furthermore, although the apparatuses in FIG. 9 and FIG. 10A throughFIG. 10C are shown with image establishers that establish twodimensional images and model determiners that determine threedimensional models therefrom, this is an example only. As previouslynoted with regard to methods according to the present invention, forcertain embodiments three dimensional models may be established withoutnecessarily establishing and/or considering two dimensional images. Insuch instances, an apparatus according to the present invention may notinclude learning, gallery, and query image establishers, or may includeonly a query image determiner for determining a two dimensional queryimage (e.g. for comparison with transformed gallery images) from a threedimensional query model, etc. Similarly, embodiments that do notdetermine a pre-transform may not include a pre-transform determiner.These are examples only, and other arrangements may be equally suitable.

Although the present invention as shown and described with regard toFIG. 9, FIG. 10A through FIG. 10C, and FIG. 11A through FIG. 11C may beconsidered to have the respective data entities and/or executableinstructions and/or data making up data entities already instantiatedupon the respective processors, the present invention is not so limited.For at least certain embodiments, data entities may be read from a datastore, received via a communicator, etc. and instantiated onto theprocessor(s). That is, a learning image establisher, a learning modeldeterminer, a gallery image establisher, a gallery model determiner, aquery image establisher, a query model determiner, a learning modelselector, a pre-transform determiner, a transform determiner, a modeltransformer, a transformed gallery image determiner, an image comparer,etc. may be instantiated onto one or more processors.

The above specification, examples, and data provide a completedescription of the manufacture and use of the composition of theinvention. Since many embodiments of the invention can be made withoutdeparting from the spirit and scope of the invention, the inventionresides in the claims hereinafter appended.

We claim:
 1. A method, comprising: establishing at least onesubstantially three dimensional learning model of at least one learningsubject; establishing at least one substantially three dimensionalgallery model for at least one gallery subject; establishing at leastone substantially three dimensional query model of a query subject;determining a transform of at least one parent gallery model from amongsaid at least one gallery model in combination with at least one activelearning model from among said at least one learning model so as toyield at least one transformed gallery model, wherein said transformedgallery model approaches correspondence with at least one of said atleast one query model in at least one model property as compared withsaid parent gallery model; applying said transform; and comparing atleast one substantially two dimensional transformed gallery image atleast substantially corresponding with said at least one transformedgallery model against at least one substantially two dimensional queryimage at least substantially corresponding with said at least one querymodel, so as to determine whether said at least one query subject issaid at least one gallery subject.
 2. The method of claim 1, wherein:each of said learning images comprises a unique state of imageproperties as compared with a remainder of said learning images.
 3. Themethod of claim 2, wherein: said image properties comprise at least oneof viewing aspect, illumination, texture, and configuration.
 4. Themethod of claim 1, wherein: each of said gallery images comprises atleast one substantially similar image property as compared with aremainder of said gallery images.
 5. The method of claim 4, wherein:said at least one image property comprises at least one of viewingaspect, illumination, texture, and configuration.
 6. The method of claim1, wherein: each of said query images comprises a unique state of imageproperties as compared with said gallery images.
 7. The method of claim6, wherein: said image properties comprise at least one of viewingaspect, illumination, texture, and configuration.
 8. The method of claim1, comprising: determining said at least one transformed gallery imagefrom said at least one transformed gallery model.
 9. The method of claim1, comprising: determining said at least one query image from said atleast one query model.
 10. The method of claim 1, wherein: establishingsaid at least one learning model comprises at least one of laserscanning, three dimensional tomography, time-of-flight measurement,depth imaging, ultrasonic mapping, holographic imaging, and plenopticphotography.
 11. The method of claim 1, wherein: establishing said atleast one gallery model comprises at least one of laser scanning, threedimensional tomography, time-of-flight measurement, depth imaging,ultrasonic mapping, holographic imaging, and plenoptic photography. 12.The method of claim 1, wherein: establishing said at least one querymodel comprises at least one of laser scanning, three dimensionaltomography, time-of-flight measurement, depth imaging, ultrasonicmapping, holographic imaging, and plenoptic photography.
 13. The methodof claim 1, comprising: establishing at least one substantially twodimensional learning image of said at least one learning subject; anddetermining said at least one learning model therefrom.
 14. The methodof claim 13, wherein: establishing said at least one learning imagecomprises at least one of digital photography, analog photography, twodimensional scanning, visible light imaging, near infrared imaging,thermal infrared imaging, ultraviolet imaging, monocrhome imaging, colorimaging, multispectral imaging, hyperspectral imaging, millimeter waveimaging, transmissive x-ray imaging, and backscatter x-ray imaging. 15.The method of claim 1, comprising: establishing at least onesubstantially two dimensional gallery image of said at least one gallerysubject; and determining said at least one gallery model therefrom. 16.The method of 15, wherein: establishing said at least one gallery imagecomprises at least one of digital photography, analog photography, twodimensional scanning, visible light imaging, near infrared imaging,thermal infrared imaging, ultraviolet imaging, monocrhome imaging, colorimaging, multispectral imaging, hyperspectral imaging, millimeter waveimaging, transmissive x-ray imaging, and backscatter x-ray imaging. 17.The method of claim 1, comprising: establishing at least onesubstantially two dimensional query image of said at least one querysubject; and determining said at least one query model therefrom. 18.The method of claim 17, wherein: establishing said at least one queryimage comprises at least one of digital photography, analog photography,two dimensional scanning, visible light imaging, near infrared imaging,thermal infrared imaging, ultraviolet imaging, monocrhome imaging, colorimaging, multispectral imaging, hyperspectral imaging, millimeter waveimaging, transmissive x-ray imaging, and backscatter x-ray imaging. 19.The method of claim 1, wherein: said at least one learning subjectcomprises a human face; said at least one gallery subject comprises ahuman face; and said at least one query subject comprises a human face.20. The method of claim 1, wherein: said at least one learning subjectcomprises at least one of a human, an animal, a plant, a landscapefeature, a vehicle, a weapon, a food item, and a tool; said at least onegallery subject comprises said at least one of a human, an animal, aplant, a landscape feature, a vehicle, a weapon, a food item, and atool; said at least one query subject comprises said at least one of ahuman, an animal, a plant, a landscape feature, a vehicle, a weapon, afood item, and a tool;
 21. The method of claim 1, comprising:determining a pre-transform of at least one parent query model fromamong said at least one query model in combination with at least oneactive learning model from among said at least one learning model so asto yield at least one transformed query model, wherein said transformedquery model approaches correspondence with at least one of said at leastone gallery model in at least one model property as compared with saidparent query model; and determining said transform as being at leastsubstantially an inverse of said pre-transform.
 22. The method of claim1, wherein: said transform is a three dimensional transform.
 23. Themethod of claim 1, wherein: said transform comprises a three dimensionalillumination transfer.
 24. The method of claim 1, wherein: saidtransform comprises a three dimensional aspect alignment.
 25. The methodof claim 1, wherein: said transform comprises a three dimensionalreconfiguration.
 26. The method of claim 1, wherein: said at least onemodel property comprises at least one of texture, shape, illumination,and configuration.
 27. A method, comprising: establishing at least onesubstantially three dimensional learning model of at least one learningsubject; establishing at least one substantially three dimensionalgallery model for at least one gallery subject; establishing at leastone substantially three dimensional query model of a query subject;determining a transform of at least one parent query model from amongsaid at least one query model in combination with at least one activelearning model from among said at least one learning model so as toyield at least one transformed query model, wherein said transformedquery model approaches correspondence with at least one of said at leastone gallery model in at least one model property as compared with saidparent query model; applying said transform; comparing at least onesubstantially two dimensional transformed query image at leastsubstantially corresponding with said at least one transformed querymodel against at least one substantially two dimensional gallery imageat least substantially corresponding with said at least one gallerymodel, so as to determine whether said at least one query subject issaid at least one gallery subject.
 28. A method, comprising:establishing at least one substantially three dimensional learning modelof at least one learning subject; establishing at least onesubstantially three dimensional gallery model for at least one gallerysubject; establishing at least one substantially three dimensional querymodel of a query subject; determining a transform of at least one parentgallery model from among said at least one gallery model in combinationwith at least one active learning model from among said at least onelearning model so as to yield at least one transformed gallery model,wherein said transformed gallery model approaches correspondence with atleast one of said at least one query model in at least one modelproperty as compared with said parent gallery model; applying saidtransform; comparing said at least one transformed gallery model againstsaid at least one query model, so as to determine whether said at leastone query subject is said at least one gallery subject.
 29. A method,comprising: establishing at least one substantially three dimensionallearning model of at least one learning subject; establishing at leastone substantially three dimensional gallery model for at least onegallery subject; establishing at least one substantially threedimensional query model of a query subject; determining a transform ofat least one parent query model from among said at least one query modelin combination with at least one active learning model from among saidat least one learning model so as to yield at least one transformedquery model, wherein said transformed query model approachescorrespondence with at least one of said at least one gallery model inat least one model property as compared with said parent query model;applying said transform; comparing said at least one transformed querymodel against said at least one gallery model, so as to determinewhether said at least one query subject is said at least one gallerysubject.
 30. A method, comprising: capturing a plurality of twodimensional digital learning images of a learning face, each of saidlearning images comprising a unique state of viewing aspect,illumination, texture, and configuration as compared with a remainder ofsaid learning images; determining computationally a three dimensionallearning model from said learning images; capturing a plurality of twodimensional digital gallery images, one gallery image from each of aplurality of gallery faces, each of said gallery images comprising astate of at least substantially similar viewing aspect, illumination,and configuration as compared with a remainder of said gallery images;determining computationally a plurality of three dimensional gallerymodels from said gallery images, one for each of said plurality ofgallery faces; capturing a two-dimensional query image of a query face,said query image comprising a state of viewing aspect, illumination, andconfiguration at least substantially different from any of said galleryimages; determining computationally a three dimensional query model fromsaid query image; determining for each of said gallery models apre-transform of said query model in combination with said learningmodel so as to yield a transformed query model, wherein each transformedquery model approaches correspondence with regard to at least one oftexture, shape, illumination, and configuration with a respective one ofsaid gallery models, as compared with said query model; determining foreach of said gallery models a transform as being at least substantiallyan inverse of said respective pre-transform therefor; applying saidtransforms to said respective gallery models so as to yield transformedgallery models; determining computationally a two dimensionaltransformed gallery image from each of said transformed gallery models;comparing each of said transformed gallery images against said queryimage so as to determine whether said at least one query subject is anyof said gallery subjects.
 31. An apparatus, comprising: a processor; atleast one of a sensor, a data store, and a communicator, incommunication with said processor; a learning image establishercomprising executable instructions instantiated on said processor, saidlearning image establisher being adapted to establish at least one twodimensional learning image of at least one learning subject via said atleast one of said at least one of said sensor, said data store, and saidcommunicator; a learning model determiner comprising executableinstructions instantiated on said processor, said learning modeldeterminer being adapted to determine at least one three dimensionallearning model from said at least one learning image; a gallery imageestablisher comprising executable instructions instantiated on saidprocessor, said gallery image establisher being adapted to establish atleast one two dimensional gallery image of at least one gallery subjectvia said at least one of said at least one of said sensor, said datastore, and said communicator; a gallery model determiner comprisingexecutable instructions instantiated on said processor, said gallerymodel determiner being adapted to determine at least one threedimensional gallery model from said at least one gallery image; a queryimage establisher comprising executable instructions instantiated onsaid processor, said query image establisher being adapted to establishat least one two dimensional query image of at least one query subjectvia said at least one of said at least one of said sensor, said datastore, and said communicator; a query model determiner comprisingexecutable instructions instantiated on said processor, said query modeldeterminer being adapted to determine at least one three dimensionalquery model from said at least one query image; a learning modelselector comprising executable instructions instantiated on saidprocessor, said learning model selector being adapted to select at leastone active learning model from said at least one learning model; apre-transform determiner comprising executable instructions instantiatedon said processor, said pre-transform determiner being adapted todetermine a pre-transform of at least one parent query model from amongsaid at least one query model in combination with at least one activelearning model from among said at least one learning model so as toyield at least one transformed query model, wherein said transformedquery model approaches correspondence with at least one of said at leastone gallery model in at least one model property as compared with saidparent query model; a transform determiner comprising executableinstructions instantiated on said processor, said transform determinerbeing adapted to determine said transform as being at leastsubstantially an inverse of said pre-transform; a model transformercomprising executable instructions instantiated on said processor, saidmodel transformer being adapted to transform said at least one gallerymodel to yield at least one transformed gallery model; a transformedgallery image determiner comprising executable instructions instantiatedon said processor, said transformed gallery image determiner beingadapted to determine at least one two dimensional transformed galleryimage from said at least one transformed gallery model; and an imagecomparer comprising executable instructions instantiated on saidprocessor, said image comparer being adapted to compare said at leastone transformed gallery image against said at least one query image soas to determine whether said at least one query subject is said at leastone gallery subject.
 32. An apparatus, comprising: a processor; a sensorin communication with said processor, said sensor being adapted to sensetwo dimensional images; at least one of a data store and a communicator,in communication with said processor; a learning model establishercomprising executable instructions instantiated on said processor, saidlearning model establisher being adapted to establish at least one threedimensional learning model of at least one learning subject via said atleast one of said at least one of said data store and said communicator;a gallery model establisher comprising executable instructionsinstantiated on said processor, said gallery model establisher beingadapted to establish at least one three dimensional gallery image of atleast one gallery subject via said at least one of said at least one ofsaid data store and said communicator; a query image establishercomprising executable instructions instantiated on said processor, saidquery image establisher being adapted to establish at least one twodimensional query image of at least one query subject via said sensor; aquery model determiner comprising executable instructions instantiatedon said processor, said query model determiner being adapted todetermine at least one three dimensional query model from said at leastone query image; a learning model selector comprising executableinstructions instantiated on said processor, said learning modelselector being adapted to select at least one active learning model fromsaid at least one learning model; a pre-transform determiner comprisingexecutable instructions instantiated on said processor, saidpre-transform determiner being adapted to determine a pre-transform ofat least one parent query model from among said at least one query modelin combination with at least one active learning model from among saidat least one learning model so as to yield at least one transformedquery model, wherein said transformed query model approachescorrespondence with at least one of said at least one gallery model inat least one model property as compared with said parent query model; atransform determiner comprising executable instructions instantiated onsaid processor, said transform determiner being adapted to determinesaid transform as being at least substantially an inverse of saidpre-transform; a model transformer comprising executable instructionsinstantiated on said processor, said model transformer being adapted totransform said at least one gallery model to yield at least onetransformed gallery model; a transformed gallery image determinercomprising executable instructions instantiated on said processor, saidtransformed gallery image determiner being adapted to determine at leastone two dimensional transformed gallery image from said at least onetransformed gallery model; and an image comparer comprising executableinstructions instantiated on said processor, said image comparer beingadapted to compare said at least one transformed gallery image againstsaid at least one query image so as to determine whether said at leastone query subject is said at least one gallery subject.
 33. A headmounted display, comprising: a processor; a sensor in communication withsaid processor, said sensor being adapted to sense two dimensionalimages; at least one of a data store and a communicator, incommunication with said processor; a learning model establishercomprising executable instructions instantiated on said processor, saidlearning model establisher being adapted to establish a threedimensional learning model of a learning subject via said at least oneof said at least one of said data store and said communicator; a gallerymodel establisher comprising executable instructions instantiated onsaid processor, said gallery model establisher being adapted toestablish a three dimensional gallery image of each of a plurality ofgallery subjects via said at least one of said at least one of said datastore and said communicator; a query image establisher comprisingexecutable instructions instantiated on said processor, said query imageestablisher being adapted to establish a two dimensional query image ofa query subject via said sensor; a query model determiner comprisingexecutable instructions instantiated on said processor, said query modeldeterminer being adapted to determine a three dimensional query modelfrom said query image; a pre-transform determiner comprising executableinstructions instantiated on said processor, said pre-transformdeterminer being adapted to determine a pre-transform of said querymodel in combination with said learning model so as to yield at leastone transformed query model approaching correspondence with said gallerymodels in at least one of texture, shape, illumination, andconfiguration as compared with said parent query model; a transformdeterminer comprising executable instructions instantiated on saidprocessor, said transform determiner being adapted to determine at leastone transform as at least substantially an inverse of said at least onepre-transform; a model transformer comprising executable instructionsinstantiated on said processor, said model transformer being adapted totransform said gallery models to yield a plurality of transformedgallery models; a transformed gallery image determiner comprisingexecutable instructions instantiated on said processor, said transformedgallery image determiner being adapted to determine two dimensionaltransformed gallery images from said transformed gallery models; animage comparer comprising executable instructions instantiated on saidprocessor, said image comparer being adapted to compare said transformedgallery images against said query image so as to determine whether querysubject is any of said gallery subjects; and an outputter incommunication with said processor, said outputter being adapted tooutput visual content regarding a comparison result as to whether saidquery subject is any of said gallery subjects; said processor, saidsensor, said at least one of said data store and said communicator, andsaid outputter being disposed on a frame, said frame being configured soas to be wearable on the head of a wearer, wherein when said frame isworn said outputter is disposed proximate, facing, and substantiallyaligned with at least one eye of said wearer, and said sensor isdisposed so as to at least substantially match a line of sight of atleast one eye of said wearer.
 34. An apparatus, comprising: means forestablishing at least one substantially three dimensional learning modelof at least one learning subject; means for establishing at least onesubstantially three dimensional gallery model for at least one gallerysubject; means for establishing at least one substantially threedimensional query model of a query subject; means for determining atransform of at least one parent gallery model from among said at leastone gallery model in combination with at least one active learning modelfrom among said at least one learning model so as to yield at least onetransformed gallery model, wherein said transformed gallery modelapproaches correspondence with at least one of said at least one querymodel in at least one model property as compared with said parentgallery model; means for applying said transform; and means forcomparing at least one substantially two dimensional transformed galleryimage at least substantially corresponding with said at least onetransformed gallery model against at least one substantially twodimensional query image at least substantially corresponding with saidat least one query model, so as to determine whether said at least onequery subject is said at least one gallery subject.